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1.  INTRODUCTION 


The  overall  goals  of  this  research  program  are  to  improve  tactical  combat  casualty  care  (TCCC)  training 
and  to  develop  a  principled  approach  to  designing  augmented  reality  training  for  combat  medics.  As  a 
first  step,  this  Phase  I  STTR  focused  on  determining  the  technical  feasibility  of  developing  an  augmented 
reality  (AR)  combat  medic  training  system  that  incorporates  adaptive  training  principles.  During  this 
Phase  I  effort,  Unveil,  LLC  sought  to  integrate  AR  with  smart  manikins  to  create  learning  experiences 
that  are  scenario-based  and  interactive.  We  developed  scenarios  that  train  and  assess  the  macrocognitive 
skills  of  sensemaking  (e.g.,  diagnosing  patient  conditions),  problem  detection,  cue  recognition,  and 
treatment  planning  in  dynamic  environments.  The  objectives  of  this  project  were:  1)  overcome  the 
technological  barriers  to  mapping  AR  images  onto  physical  manikins;  2)  develop  prototype  AR-based 
training;  3)  create  two  training  scenarios  (tension  pneumothorax  and  airway  obstruction);  and  4)  design  a 
pilot  study  that  integrates  instructional  system  design  principles  and  macrocognitive  measures.  The 
resulting  AR-based  adaptive  training  platform  will  be  the  foundation  for  a  series  of  studies  that  will 
inform  an  evidence-based  set  of  principles  to  guide  the  design  and  use  of  AR-based  training  in  Phase  II 
and  beyond. 

2.  OVERALL  PROJECT  SUMMARY 

TCCC  is  a  military  training  approach  that  focuses  on  treating  the  most  common  fatal  (but  treatable) 
battlefield  injuries  -  massive  hemorrhage,  tension  pneumothorax,  and  airway  obstruction.  TCCC  is 
effective  -  since  it  was  implemented,  combat  deaths  from  these  injuries  have  declined  (Holcomb,  2009; 
Gresham  &  Rodriguez,  2005-2006;  Bottoms,  2006).  Part  of  the  Joint  Trauma  System  (JTS;  DoDI  6040.47 
Joint  Trauma  System,  2016),  current  TCCC  training  focuses  on  the  skills  required  to  treat  these  types  of 
injuries,  but  not  the  perceptual  cues  and  mental  models  needed  to  quickly  size  up  a  situation  and  react 
appropriately.  Furthermore,  a  retrospective  chart  review  of  patients  who  received  pre-hospital 
cricothyroidotomy  during  combat  operations  in  Iraq  and  Afghanistan  between  September  2007  and  July 
2009  revealed  that  only  68%  of  the  procedures  were  successful,  suggesting  that  there  may  be  training 
gaps  associated  with  assessing  the  effectiveness  of  some  medical  procedures  in  the  field  (Mabry  & 
Frankfurt,  2012).  Knowing  how  to  do  a  procedure  is  only  useful  when  coupled  with  the  knowledge  of 
when  the  procedure  must  be  used,  and  how  to  monitor  whether  it  has  been  effective.  We  believe  that  well- 
designed  AR  training  can  help  fill  these  gaps  in  the  TCCC  training  curriculum. 

During  the  course  of  this  Phase  I  STTR,  Unveil  advanced  its  existing  training  system  prototype  to  better 
meet  the  needs  of  the  combat  medic  community.  Our  product,  the  Macrocognitive  Augmented  Reality 
Trainer  (MART),  consists  of  an  AR  headset,  worn  by  the  student,  a  smart  manikin,  and  an  instructor 
tablet.  For  this  project,  we  used  the  Microsoft  HoloLens  AR  headset,  the  Tactical  Operations  Medical 
Manikin  (TOMManikin),  and  a  tablet-based  app  for  the  instructor  interface.  The  software  developed 
during  this  project  is  hardware- agnostic. 


4 


Figure  1 .  MART  platform  ecosystem,  including  an  AR  headset,  smart  manikin,  and  instructor  tablet  app 


The  overarching  goal  of  a  Phase  I  STTR  is  to  establish  the  technical  merit,  feasibility,  and  commercial 
potential  of  a  proposed  idea.  To  establish  feasibility,  the  specific  objectives  of  the  current  STTR  project 
were  to: 

1 .  Overcome  technological  barriers  to  mapping  AR  images  onto  physical  manikins 

2.  Develop  prototype  AR-based  training 

3.  Create  training  and  evaluation  scenarios  around  two  common  battlefield  conditions: 

a.  Tension  pneumothorax 

b.  Airway  obstruction 

4.  Design  a  pilot  study  to  test  the  AR  training  technology 

We  articulated  7  tasks  required  to  achieve  these  objectives.  Below,  we  list  the  major  activities  and 
accomplishments  associated  with  each  of  these  tasks.  Then  we  provide  additional  details  about  methods, 
approach,  and  outcomes  for  each  task  within  the  context  of  the  project  objectives. 

Project  Tasks 


Task  1:  Kickoff  and  IRB  submission 

We  held  an  internal  kickoff  to  orient  the  entire  team  to  the  project  goals,  objectives,  and  tasks.  Unveil  also 
traveled  to  the  Office  of  Naval  Research  to  attend  a  kickoff  meeting  with  the  contracting  officer’s 
representative  (COR),  Dr.  Ray  Perez  and  other  ONR  personnel.  The  meeting  provided  insight  into  ONR 
priorities  and  informed  detailed  project  planning. 

Task  2:  Scenario  development 

We  interviewed  four  emergency  medicine  subject  matter  experts  (SMEs)  about  tension  pneumothorax  and 
airway  obstruction  cases  they  had  experienced.  We  used  exemplars  from  these  interviews  to  draft  two 
training  scenarios  (one  for  tension  pneumothorax,  one  for  airway  obstruction).  We  sent  the  draft  scenarios 
to  the  SMEs  to  review  and  respond  to,  then  revised  based  on  their  feedback.  Scenarios  are  discussed  in 
more  detail  in  the  description  of  Objective  3  below. 

Accomplishments  associated  with  Task  2  were: 

•  One  tension  pneumothorax  training  scenario 

•  One  airway  obstruction  training  scenario 

Task  3:  Expert  model  development 

When  the  SMEs  reviewed  the  draft  training  scenarios,  they  also  responded  to  a  series  of  cognitive  probes 
designed  to  help  them  articulate  how  they  would  assess,  treat,  and  plan  for  the  patient  in  each  scenario. 


5 


We  integrated  responses  from  SMEs  to  create  an  expert  model  for  each  of  the  two  training  scenarios. 
Team  member  Oliver  Smith.  Pararescue  Jumper  (PJ)  and  Combat  Rescue  Officer  (CRO),  refined  the 
integrated  expert  models.  The  expert  models  are  discussed  in  more  detail  in  the  description  of  Objective  3 
below. 

Accomplishments  associated  with  Task  3  were: 

•  Expert  model  for  tension  pneumothorax  training  scenario 

•  Expert  model  for  airway  obstruction  training  scenario 

Task  4:  Prototype  development 

There  were  three  main  efforts  associated  with  the  MART  prototype  development  that  we  pursued 
simultaneously.  One  effort  was  to  develop  the  virtual  patient  and  virtual  props  to  match  the  training 
scenarios.  A  second  effort  was  to  investigate  new  technologies  to  improve  virtual  patient  registration  (i.e., 
“anchoring”  the  virtual  patient  to  a  physical  manikin,  so  they  move  together).  The  third  effort  was  to 
develop  a  plan  for  incorporating  adaptive  training  components  into  MART.  Each  of  these  development 
activities  are  discussed  in  more  detail  in  the  descriptions  for  Objective  1  and  Objective  2  below. 

Accomplishments  associated  with  Task  4  were: 

•  Virtual  patient  registration  proposed  solutions 

•  Virtual  patient  with  virtual  props  to  match  training  scenarios 

•  Description  of  adaptive  training  components 

Task  5:  Curricula  integration 

Integration  into  training  curricula  is  critical  for  commercialization  of  the  proposed  training  technology. 
For  Task  5,  we  analyzed  existing  TCCC  training,  along  with  emergency  medicine  physician  training 
(specifically  at  Ohio  State  University  Medical  Center)  to  identify  strategies  for  integrating  MART.  We 
reviewed  the  Accreditation  Council  for  Graduate  Medical  Education  (ACGME)  program  accreditation 
requirements  and  met  with  training  leaders  and  supervisors.  We  also  observed  simulation-based  training 
(advanced  airway  management)  at  the  Ohio  State  University  Medical  Center  to  see  how  standard  training 
sessions  operate.  Additional  details  about  the  activities  related  to  curricula  integration  are  discussed  in  the 
description  of  Objective  2  below. 

The  accomplishment  associated  with  Task  5  was: 

•  Curricula  integration  plan 

Task  6:  Study  design 

For  Task  6,  we  first  reviewed  literature  related  to  evaluating  and  measuring  macrocognitive  skills.  We 
also  reviewed  literature  related  to  instructional  systems  design  and  evaluating  training  programs.  The 
team  articulated  hypotheses  about  how  about  how  AR  could  be  used  to  support  adaptive  training  and 
identified  strategies  for  testing  the  strengths  and  limitations  of  individual  adaptive  training  components. 
Additional  details  are  discussed  in  the  description  of  Objective  4  below. 

Accomplishments  associated  with  this  task  were: 

•  Study  plan 

•  Evaluation  measures 


Project  Objectives 

The  seven  tasks  articulated  in  the  previous  section  supported  each  of  the  four  project  objectives.  In  this 
section,  we  describe  our  research  activities,  results  and  outcomes,  and  discuss  each  objective  in  terms  of 
the  overall  research  goal  of  creating  evidence-based  principles  about  the  design  and  use  of  AR  training 
and  applying  those  principles  to  improve  TCCC  training. 
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Objective  1:  Mapping  the  Virtual  Patient  to  a  Physical  Manikin 

Our  first  objective  was  to  develop  a  technological  approach  to  integrating  the  virtual  patient  with  a 
physical  manikin.  Physical  manikins  are  well-suited  to  support  practice  at  applying  interventions  (i.e., 
tourniquet,  needle  decompression,  cricothyroidotomy,  etc.),  but  have  limited  ability  to  depict  the 
changing  condition  of  a  patient  (i.e.,  changes  in  skin  tone,  mental  status,  injury  progression).  For  this 
project,  we  partnered  with  Innovative  Tactical  Training  Solutions  (ITTS),  integrating  the  virtual  patient 
with  ITTS’s  TOMManikin  to  fully  capitalize  on  the  strengths  of  AR  to  present  perceptual  cues  and 
provide  a  more  seamless  training  experience. 

The  challenge  was  to  develop  a  registration  process  that  would  anchor  the  virtual  patient  to  a  physical 
object.  This  allows  the  virtual  patient  to  line  up  with  the  physical  manikin,  and  to  correlate  movements  in 
the  physical  manikin  with  movements  in  the  virtual  patient  (i.e.,  if  the  manikin’s  arm  moves,  so  does  the 
virtual  patient’s  arm).  We  anticipate  that  finding  the  right  technology  solution  to  anchor  the  virtual  patient 
to  a  physical  manikin  will  provide  a  more  realistic  training  experience  and  improve  learner  interactions 
with  the  system.  The  Unveil  team  devoted  part  of  the  Phase  I  effort  to  researching  and  testing  a  number  of 
potential  solutions. 

Methods.  We  began  by  identifying  technologies  that  could  be  adapted  to  work  in  simulation-based 
training  environments,  that  could  be  used  in  an  easy  and  naturalistic  manner  by  the  end  user,  that  would 
be  compatible  with  our  simulation  platform  and  graphics  engine,  and  that  would  have  the  potential  to 
work  across  a  variety  of  mixed  reality  headsets  and  operating  systems. 

We  tested  different  technologies  to  determine  whether  we  could  show  a  real-time  synchronization 
between  the  location  and  orientation  of  physical  objects  to  virtual  counterparts  in  a  mixed  reality  view. 
First,  we  developed  hardware  firmware  and  software  application  programming  interfaces  (APIs)  for 
integrating  potential  registration  solutions  into  the  mixed  reality  testing  platform.  Then,  we  could  test  the 
potential  registration  solutions  in  tracking  manikin  movements  within  the  mixed  reality  testing  platform 
on  a  desktop  computer. 

Outcomes.  We  identified  three  promising  technologies  that  fit  the  required  system  parameters:  1)  real¬ 
time  skeletal  tracking  using  3D  depth  sensors,  2)  inertial  measurement  units  (IMUs)  added  to  the  physical 
manikin,  and  3)  visual  tracking  using  markers  applied  to  the  manikin.  Eventual  solutions  will  incorporate 
combinations  of  these  technologies,  depending  on  the  context  of  use.  Each  has  important  strengths 
described  in  more  detail  below. 

Skeleton  tracking  is  an  application  of  machine  vision  technologies  that  can  be  used  to  understand  and 
track  human  gestures.  Skeleton  and  hand  tracking  are  currently  used  in  support  of  Natural  User  Interfaces 
(NUIs).  Microsoft  Kinect  is  an  example  of  a  commercial  product  that  uses  skeleton  tracking.  During  the 
Phase  I,  we  tested  whether  this  technology  would  be  able  to  track  a  manikin.  We  tested  the  Microsoft 
Kinect  (Figure  2)  and  the  Orbbec  system  (Figure  3).  Microsoft  has  stopped  supporting  the  Kinect  system, 
but  Orbbec  is  currently  available  and  has  the  added  benefit  of  working  across  a  variety  of  operating 
systems.  Our  Phase  I  research  and  prototyping  efforts  demonstrated  that  the  humanoid  attributes  of  a 
medical  training  manikin  were  trackable  with  the  skeleton  fracking  technology,  making  it  a  viable 
candidate  in  support  of  dynamic  patient  registration.  Flowever,  the  3D  depth  sensors  required  for  skeleton 
fracking  limit  the  application  to  controlled  environments,  such  as  dedicated  simulation  centers.  For  videos 
of  our  technological  studies  of  skeleton  tracking  systems,  please  visit  the  following  links:  Microsoft 
Kinect  Tech  Study:  https://youtu.be/2n2ukNkFKDU,  Orbbec  Tech  Study: 
https  ://yout  u .  be/NMK4FB  sC  v3  Q . 

Inertial  measurement  units  (IMUs)  are  electronic  devices  that  measure  and  report  an  object’s  force, 
angular  rate,  and  in  some  cases,  surrounding  magnetic  field.  This  hardware  uses  a  combination  of 
gyroscopes,  accelerometers,  and  sometimes  magnetometers.  IMUs  are  placed  directly  on  the  object  being 
tracked.  During  Phase  I,  we  tested  whether  multiple  IMUs  could  be  customized  to  be  chained  and 
attached  to  the  manikin  to  track  the  manikin’s  movements  and  mirror  the  movements  in  the  virtual 
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patient.  Our  testing  and  prototyping  activities  demonstrated  that  our  customized  IMUs  can  provide  a 
robust  solution  to  tracking  a  manikin’s  limb  movements  (Figure  4).  One  potential  drawback  of  this 
approach  is  that  custom  hardware  development  and  integration  will  require  more  time  and  resources  to 
bring  to  market.  For  a  video  demonstration  of  the  technology  study  of  IMUs,  see  IMU  Tech  Study: 
https://youtu.be/VFWAsvAZA40. 

Microsoft  Kinect  Orbbec  Chained  IMUs 


Figure  2.  Microsoft  Kinect 
skeletal  tracking  tech  study 

Tech  study  link: 

https://voutu.be/2n2ukNkLKDU 


Figure  3.  Orbbec  skeletal  tracking 
tech  study 


Tech  study  link: 

https://youtu.be/NMK4LBsCv30 


Tech  study  link: 

https://youtu.be/VFWAsvAZA40 


Figure  4.  Chained  IMUs  tech  study 


Marker-based  tracking  uses  visual  patterns  or  images  to  allow  a  computer  vision  system  to  detect  and 
track  an  object  in  a  field  of  view.  For  example,  one  of  Unveil’s  other  products,  Chiron™,  utilizes  a  paper 
skull  with  distinctive  visual  patterns  to  facilitate  robust  object  detection  and  tracking  from  all  viewable 
angles. 

When  the  object  is  detected  within  the  field  of  view  of  an  AR  application,  a  3D  render  of  a  patient’s  head 
appears  in  the  AR  view  showing  signs  and  symptoms  of  a  condition  (e.g.,  head  trauma;  see  Figures  5  and 
6).  Marker-based  tracking  is  currently  supported  in  all  the  major  AR  platforms  as  a  means  to  add  virtual 
content  to  physical  objects,  lowering  the  technical  barrier  for  implementing  AR  content.  However,  a 
purely  marker-based  approach  requires  changing  the  manikin’s  outward  appearance  or  introducing 
additional  physical  props.  The  technology  does  not  work  if  the  marker  is  not  in  the  field  of  view  of  the 
computer  vision  system. 
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Figure  5.  Chiron  paper  skull  with  fiducial  markers  Figure  6.  Chiron's  AR  content  showing  a  head  injury 


Conclusions.  We  envision  using  a  combination  of  technologies  based  on  the  intended  use  case  for  the 
MART  system.  For  the  Phase  II,  we  envision  a  portable  version  that  can  be  quickly  set  up  in  any 
classroom,  simulation  facility,  or  even  field  setting.  IMUs  embedded  into  the  manikin  combined  with 
discreet  visual  markers  are  the  most  promising  approach  for  portable  use.  For  medical  training  simulation 
centers  where  students  come  to  a  dedicated  space  to  interact  with  the  system,  we  would  likely  add 
environmental  3D  depth  sensors  to  our  solution  to  capture  more  data  about  student  performance.  We 
anticipate  that  the  tight  coupling  between  virtual  content  and  physical  props  will  be  a  requirement  for  the 
type  of  skills  training  that  combat  medics  need.  Virtual  patient  registration  is  an  important  foundational 
step  for  testing  the  effectiveness  of  our  proposed  training  components. 

Objective  2:  Develop  Prototype  AR  Training 

While  Objective  1  was  focused  on  the  technological  advancement  of  MART,  Objective  2  was  focused  on 
creating  its  training  content  and  approach.  There  were  three  main  activities  associated  with  Objective  2: 
creating  virtual  content,  articulating  the  training  strategy,  and  developing  a  curricula  integration  plan.  We 
created  a  virtual  patient  to  correspond  with  our  training  scenarios  along  with  other  virtual  support  items 
such  as  diagnostic  tools  and  medical  interventions  (for  a  video  demonstrating  the  MART  Combat  Medic 
version  of  the  system,  please  visit  https://youtu.be/HjVxyiKczME).  We  also  articulated  a  strategy  for 
incorporating  adaptive  training  components  into  MART.  With  AR  technology,  learners  are  able  to 
interact  with  the  physical  environment  and  virtual  content  seamlessly.  This  functionality  affords  the 
ability  to  inject  hints  and  guidance  to  learners  as  they  work  through  training  scenarios.  Learners  can  see  or 
hear  virtual  hints  and  guidance,  immediately  practice  physical  interventions,  and  receive  instructor 
feedback  about  their  actions.  Finally,  we  articulated  a  curricula  integration  plan  for  incorporating  MART 
into  resident  physician  training  as  an  important  step  towards  commercializing  our  approach.  The  methods 
and  outcomes  associated  with  each  of  these  main  activities  are  described  in  turn. 

Virtual  Patient  and  Software  Development 

Methods.  We  created  the  virtual  patient  and  virtual  props  to  accompany  the  training  scenarios  using  an 
iterative  design  and  development  process.  We  worked  closely  with  our  combat  medicine  SME  (Oliver 
Smith,  PJ,  CRO);  he  reviewed  multiple  rounds  of  design  and  offered  feedback  regarding  the 
appropriateness  of  virtual  patient  and  objects  throughout  the  process.  We  bundled  and  deployed  the 
virtual  assets  (i.e.,  virtual  patients,  props)  to  the  HoloLens  mixed  reality  platform  at  various  stages  of 
development  so  we  could  confirm  their  medical  accuracy  and  verify  that  the  virtual  assets  maintained  a 
reasonable  level  of  graphics  performance. 

We  expanded  the  existing  programmatic  models  of  the  software  (HoloLens  mixed  reality  software,  tablet 
software)  to  support  virtual  patients  with  multi-system  TCCC  injuries.  We  also  developed  airway  TCCC 
scenarios  to  match  the  airway  training  scenario  and  other  closely  related  injuries.  Additionally,  we 
evolved  the  data  models  underlying  the  patient  conditions.  The  data  models  were  expanded  to  support 
multi-system  medical  patients  and  to  support  the  virtual  depiction  of  properly  and  improperly  applied 
medical  interventions  to  make  the  training  more  realistic. 
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Outcomes.  We  created  virtual  content  that  matches  the  parameters  specified  in  the  training  scenarios 
(virtual  patients  and  relevant  virtual  treatment  interventions;  Figures  7  and  8).  The  virtual  patients  show  a 
progression  of  injury  state  from  point  of  injury  to  death,  broken  into  discrete  stages.  The  pulse  oximeter  is 
a  virtual  prop  that  appears  on  the  patient’s  finger  to  give  blood-oxygen  saturation  levels  throughout  the 
progression  of  stages.  The  virtual  interventions  can  be  added  by  the  student  wearing  the  FIoloLens  by 
issuing  a  verbal  command  (“Show  intervention”)  or  by  the  instructor  by  tapping  on  the  intervention  from 
the  tablet’s  app. 


Pulse  oximeter  virtual  prop: 

■ 


Needle  decompression  virtual  intervention: 
Good  placement: 

l 

Bad  placement: 

\  / 

Kinked  line  Wrong  location 


Figure  7.  Tension  pneumothorax  virtual  patient  with  virtual  assessment  and  treatment  interventions 


Figure  8.  The  five  stages  of  the  airway  obstruction  training  scenario  showing  increased  swelling  and  eventual  loss  of 

airway. 


Stage  1 
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Adaptive  Training  Structure  and  Design 

Approach.  Our  approach  to  creating  MART’S  adaptive  training  components  is  based  on  findings  from 
the  Naturalistic  Decision  Making  (NDM)  and  cognitive  expertise  literatures.  Ericcson,  Charness, 

Hoffman,  and  Feltovich  (2006)  articulated  the  importance  of  learners  being  actively  involved  in 
experiencing  and  interpreting  cues  in  the  environment.  MART’S  adaptive  training  components  are 
designed  to  leverage  the  abilities  of  AR  to  strengthen  the  first-hand  experiences  of  learners  to  actively 
engage  them  in  what  they  are  learning. 

Research  shows  that  the  development  of  expertise  is  not  a  passive  process.  There  arc  several  elements  that 
help  learners  develop  expert  knowledge.  For  example,  exposure  to  an  expert  model  has  been  shown  to 
accelerate  skill  acquisition  (Hoffman  et  al.,  2014;  Klein  &  Borders,  2016;  Pliske,  McCloskey,  &  Klein, 
2001).  Another  element  that  has  been  demonstrated  to  differentiate  experts  from  non-experts  is  reflection 
(Ericcson,  2004;  Ericcson  et  al.,  2006).  People  who  become  experts  reflect  on  their  experiences  and 
speculate  how  situations  may  have  gone  differently.  They  develop  their  own  lessons  learned  that  they  can 
apply  to  novel  situations.  Research  also  shows  that  skill  acquisition  is  better  in  learners  who  are  motivated 
to  learn,  engaging  in  “deliberate  practice”  (Ericcson,  Krampe,  &  Tesch-Romer,  1993;  Ericcson,  2006; 
Ericsson  et  al.,  2006).  Each  of  these  constructs  (i.e.,  expert  model,  reflection,  deliberate  practice)  is 
incorporated  into  our  proposed  adaptive  training  components. 

The  process  of  learning  new  content  and  skills  can  be  helped  by  the  structure  of  the  instruction  itself. 
Research  has  shown  the  critical  importance  of  feedback  in  skill  acquisition  (Ericcson,  Krampe,  &  Tesch- 
Romer,  1993).  With  a  well-designed  AR  system,  we  can  incorporate  feedback  to  the  learners  when  it  is 
most  beneficial,  both  in  real  time  and  as  summary  feedback,  or  “knowledge  of  results”  (Xeroulis  et  al., 
2007;  Wulf  &  Shea,  2004).  The  structure  of  training  and  how  much  support  is  offered  should  vary  with 
learner  skill  level.  For  example,  a  novice  might  benefit  from  real-time  guidance  and  hints  that  an 
advanced  learner  would  find  tedious.  We  will  incorporate  scaffolding  principles  (Wilson  &  Devereux, 
2014)  into  MART’S  adaptive  training  design.  Scaffolding  is  a  technique  to  move  learners  progressively 
toward  stronger  understanding  and  greater  independence  in  the  learning  process,  teaching  learners  how  to 
direct  their  own  attention  and  actions. 

Outcomes.  We  articulated  a  structure  for  the  adaptive  training  component  of  our  system.  We  envision 
three  classes  of  learner:  early  novice  (Level  1),  intermediate  learner  (Level  2),  and  advanced  learner 
(Level  3).  The  system  will  behave  differently  depending  on  the  learner’s  level,  using  different 
combinations  of  training  components.  We  articulated  training  components  for  both  the  student  and  the 
instructor. 

Learner-Facing  Adaptive  Training  Components.  We  identified  5  learner-facing  adaptive  training 
components  to  be  developed  and  evaluated  in  Phase  II.  The  first  is  the  use  of  attention-directing  hints.  In 
the  simplest  form,  this  will  appear  as  a  voice  avatar  representing  a  more  experienced  medic  (or  physician) 
encouraging  the  learner  to  notice  or  look  for  something  specific  (i.e..  Is  he  bleeding  anywhere  else?  Have 
you  looked  inside  his  mouth  to  see  if  the  airway  is  swelling?).  This  approach  supports  the  learner  to 
obtaining  first-hand  experiences  in  the  context  of  challenging  cases.  Aimed  at  novice  and  intermediate 
learners,  this  approach  provides  important  scaffolding,  and  also  retains  important  realism  by  using  a  voice 
avatar  of  a  mentor/colleague. 

Another  type  of  attention-directing  hint  we  will  explore  is  the  use  of  visual  pointers  (Figure  9).  A  visual 
cue  might  point  to  an  appropriate  intervention.  As  compared  to  a  voice  avatar  directing  the  learner  to  look 
for  signs  of  airway  swelling  or  describing  appropriate  needle  placement,  the  pointer  on  the  anatomy 
reduces  ambiguity  for  the  learner,  making  it  most  appropriate  for  novices. 
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Figure  9.  Visual  pointer  showing  where  an  important  cue  is  located  (left),  and  a  visual  cue  guiding  the  learner  to  an 

appropriate  intervention  (right) 

A  second  adaptive  training  component  is  real-time  auditory  correction,  intended  to  aid  the  learner  in 
conducting  a  thorough  assessment  using  a  relevant  algorithm.  This  training  component  incorporates  real¬ 
time  feedback  into  the  training  scenario.  For  example,  many  medical  schools  use  the  Airway,  Breathing, 
Circulation,  Disability,  Expose  the  patient,  Foley  and  rectal  (ABCDEF)  algorithm,  whereas  combat 
medics  use  the  Massive  hemorrhage,  Airway,  Respirations,  Cardiac,  Head  injury/Hypothermia  (MARCH) 
algorithm.  When  the  learner  deviates  from  the  appropriate  algorithm  (i.e.,  delays  assessing  airway 
because  of  distracting  head  injury),  an  auditory  correction  will  remind  the  learner  to  walk  through  the 
assessment  in  the  correct  order.  This  intervention  is  aimed  at  novice  and  intermediate  learners. 

A  third  adaptive  training  component  is  the  use  of  3D  animation  to  show  underlying  dynamic  anatomy. 
Although  a  learner  may  have  read  descriptions  of  what  happens  inside  the  chest  cavity  when  a  tension 
pneumothorax  develops,  seeing  an  animation  juxtaposed  with  the  physical  appearance  of  the  patient  may 
have  added  training  benefit.  By  situating  instructional  animations  of  injury  processes  in  realistic 
scenarios,  our  training  will  aid  learners  in  building  more  robust  mental  models  and  improve  their  ability 
to  visualize  the  implications  of  the  injury,  anticipate  future  states,  and  identify  appropriate  treatment 
options.  This  intervention  is  aimed  at  novice  and  intermediate  learners. 

A  fourth  adaptive  training  component  is  to  manipulate  time  using  AR.  It  will  be  possible  to  show  how  the 
patient  condition  degrades  over  time  without  medical  intervention  by  compressing  time  scales.  The 
learner  may  see  the  virtual  patient  immediately  following  an  injury,  and  then  at  whatever  time  intervals 
are  required  to  demonstrate  the  progression  of  an  injury.  For  a  tension  pneumothorax,  this  might  jump 
forward  several  hours,  while  an  airway  injury  might  unfold  in  a  matter  of  minutes.  The  learner  can  also 
jump  back  and  forth  in  time  to  compare  cue  representation.  For  example,  the  learner  might  first  notice 
jugular  vein  distention  (JVD)  at  Stage  4  of  a  scenario,  but  wonder  if  s/he  could  have  noticed  it  earlier.  The 
learner  could  then  flip  back  and  forth  between  Stage  2,  Stage  3,  and  Stage  4  to  see  how  the  JVD  initially 
appeared,  and  how  far  it  progressed.  This  component  supports  feedback  and  first-hand  experiences. 

The  performance  summary  is  the  fifth  adaptive  training  component,  made  up  of  three  parts:  expert  model, 
replay,  and  performance  score.  For  intermediate  and  advanced  learners,  the  performance  summary  will 
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highlight  where  student  performance  deviated  from  the  expert  model  based  on  gaze  tracking  data  and 
instructor  input.  It  will  be  possible  to  revisit  portions  of  the  scenario  from  the  perspective  of  the  learner. 
This  will  allow  the  learner  to  review  cues  that  were  missed  or  misinterpreted.  Finally,  learners  will 
receive  a  performance  score  in  which  points  are  awarded  for  appropriate  actions,  and  subtracted  for  hints 
used  and  errors.  The  integration  of  these  features  into  a  performance  summary  will  support  summary 
feedback,  reflection,  and  deliberate  practice. 

Instructor-Facing  Adaptive  Training  Components.  To  support  the  trainer  in  using  the  AR-based  adaptive 
training  system  we  have  four  instructor-facing  components.  To  track  perception  and  recognition,  the  gaze 
tracking  feature  allows  the  trainer  to  see  where  the  learner  is  looking.  Furthermore,  gaze  tracking  data  can 
be  compared  to  the  expert  model  to  highlight  gaps  and  errors.  If  the  learner  has  not  yet  looked  at  the  pulse 
oximeter,  for  example,  the  instructor-facing  software  will  highlight  this  omission  (see  Figure  10). 


Figure  10.  Instmctor  tablet  view  showing  gaze  tracking  results.  Green  checks  indicate  cues  that  the  student  has 

looked  at,  and  red  Xs  are  cues  that  have  been  missed. 

To  aid  in  applying  appropriate  scaffolding,  the  instructor-facing  app  will  highlight  common  errors,  as 
well  as  specific  errors  made  by  the  learner,  and  suggest  scaffolding  techniques  at  relevant  points  in  the 
scenario.  For  example,  the  software  will  suggest  a  specific  hint  if  the  learner  seems  to  have  missed  an 
important  cue. 

To  support  the  trainer  in  evaluating  student  performance,  the  instructor  interface  allows  the  trainer  to 
mark  any  part  of  the  scenario  for  later  discussion  and  replay,  such  as  notes  about  good  or  poor  technique. 

To  support  the  trainer  in  providing  constructive  feedback,  the  performance  summary  page  can  be  used  to 
highlight  differences  between  student  performance  and  the  expert  model,  access  expert  rationale  for 
specific  actions,  and  replay  specific  points  in  the  scenario. 

Curricula  Integration  Plan 

The  goal  of  this  project  is  to  improve  TCCC  training.  However,  to  take  advantage  of  partnerships  already 
in  place,  we  plan  to  test  curricula  integration  efforts  with  the  Ohio  State  University  Medical  Center.  The 
Accreditation  Council  for  Graduate  Medical  Education  (ACGME)  accredits  all  graduate  medical 
education  in  the  United  States,  including  residency  physician  training  programs  for  all  specialties.  In 
2014,  the  ACGME  mandated  the  Next  Accreditation  System  (NAS),  which  provided  increased  flexibility 
in  meeting  ACGME  competencies,  in  part  to  allow  simulation-based  medical  education.  In  many 
residency  programs,  there  is  a  procedural  skills  curriculum  with  specific  procedures  required  for  each 
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specialty.  A  common  procedural  skill  trained  in  all  programs  is  airway  management,  in  large  paid  to  avoid 
harm  from  learning  a  procedure  on  real  patients.  For  example,  the  initial  success  rate  for  first-time 
tracheal  intubation  is  between  35  and  65%  (Nouruzi-Sedeh,  Schumann,  &  Groeben,  2009),  as  compared 
to  over  90%  for  experienced  physicians.  With  a  failed  tracheal  intubation,  there  is  the  possibility  of 
perforation  or  laceration  of  the  surrounding  areas,  as  well  as  infection. 

At  Ohio  State  University,  all  resident  physicians  in  emergency  care,  trauma  care,  and  anesthesiology 
participate  in  simulation-based  training  to  meet  ACGME  requirements  for  airway  management,  which 
includes  a  tension  pneumothorax  patient.  The  training  occurs  in  the  Prior  Flail  Simulation  Center  in  the 
procedural  skills  room  with  a  part-task  training  manikin  that  includes  an  open  airway,  tongue,  and 
inflatable  balloons  that  represent  the  lungs  and  the  stomach.  Extensive  airway  management  equipment  is 
available  that  interfaces  with  the  manikin.  Cases  are  face  valid.  At  the  completion  of  the  case,  the  resident 
completes  a  self-assessment  and  an  educator  completes  an  assessment.  If  the  resident  passes,  then  the 
video  serves  as  proof  of  competency  for  ACGME  accreditation.  If  the  resident  fails,  remediation  occurs 
real-time  and  a  retake  test  is  scheduled.  In  addition,  extensive  simulation  capstone  scenarios  are  run  with 
a  full  body  manikin  that  includes  airway  management  elements  in  addition  to  other  elements.  In  this 
situation,  vital  signs  data  and  breathing  sounds  are  included  as  part  of  the  simulation,  and  the  session  is 
video-recorded. 

Representatives  from  the  OSU  residency  physician  program  expressed  an  interest  in  training  with  a 
virtual  patient  independent  of  a  physical  manikin.  The  vision  is  that  rather  than  integrating  with  a 
manikin,  the  virtual  patient  would  appear  on  the  hospital  bed  where  a  manikin  is  typically  situated.  An 
instructor  will  use  a  tablet  to  view  real-time  gaze  tracking  and  other  relevant  data  for  facilitating  the 
session.  In  order  to  accommodate  learning  in  small  groups,  they  have  asked  us  to  explore  strategies  for 
displaying  the  augmented  reality  view  simultaneously  on  a  large  monitor  in  the  simulation  room. 

Conclusions.  We  were  able  to  create  virtual  assets  (patients,  props)  and  upload  them  to  the  FIoloLens 
mixed  reality  platform  so  they  could  run  with  high  quality  graphical  resolution.  We  updated  the 
underlying  data  models  to  reflect  the  training  scenario  parameters.  We  are  working  on  streamlining  the 
process  of  creating  new  virtual  content  to  match  new  scenarios  and  injury  types.  We  began  exploring  a 
new  underlying  patient  architecture  that  would  allow  us  to  decouple  specific  parameters  (e.g.,  heart  rate, 
skin  tones,  blood  pressure,  etc.)  to  provide  near  infinite  flexibility  in  how  the  patient's  vital  signs,  cues, 
and  symptoms  are  combined  to  depict  a  medical  condition.  Instead  of  having  a  “tension  pneumothorax” 
scenario  and  a  “burned  airway”  scenario,  we  could  mix  and  max  different  injuries  and  conditions  to  create 
new  scenarios  much  more  quickly.  We  believe  this  new  approach  can  better  support  our  envisioned 
adaptive  training  framework  as  well  as  (down  the  road)  integrate  with  bio-physics  engines,  such  as 
BioGears,  to  allow  for  dynamic,  medically  accurate  simulations. 

MART  will  be  an  adaptive  training  system  in  that  it  will  behave  differently  based  on  the  skill  level  of  the 
learner.  The  presentation  of  the  first  four  learner-facing  adaptive  training  components  (attention-directing 
hints,  real-time  correction,  animation,  and  time  manipulation)  will  be  dependent  on  the  mode  selected. 

For  Level  1  learners,  MART  will  operate  in  tutorial  mode,  in  which  the  learner  is  guided  through  the 
scenarios,  and  relevant  hints,  corrections,  and  animations  will  automatically  appear  at  relevant  times  in 
the  scenario.  For  Level  2  learners,  MART  will  enter  a  high  challenge/high  support  mode  (Wilson  & 
Devereux,  2014),  in  which  hints,  corrections,  and  animations  are  available  to  learners  if  they  ask,  if  the 
instructor  thinks  they  need  additional  support,  or  if  the  system  determines  that  they  arc  going  down  an 
incorrect  path.  At  this  level,  learners  are  self-directed  through  the  scenario,  but  there  is  scaffolding  in 
place  to  ensure  they  do  not  fail.  For  Level  3  learners,  MART  enters  a  high  challenge/low  support  mode 
(Wilson  &  Devereux,  2014),  in  which  hints  are  not  available  and  the  learner  is  able  to  fail.  Discussions 
with  emergency  medicine  SMEs  with  training  experience  emphasized  the  importance  of  allowing  students 
to  fail  in  emergency  medicine  training  scenarios. 

We  have  a  plan  for  implementing  specific  components  of  the  MART  system  into  an  existing  resident 
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physician  training  program.  As  many  resident  physician  training  programs  are  already  employing 
simulation-based  training  using  manikins,  it  is  expected  that  our  university  partner,  Ohio  State  University, 
can  serve  as  a  ‘best  practice’  model  for  implementation  of  AR  for  airway  management  and  tension 
pneumothorax  training  and  have  lessons  learned  via  traditional  dissemination  pathways  to  other  academic 
medical  centers.  We  have  three  physicians  on  our  team  for  the  Phase  II  who  will  assist  in  creating  the 
practice  model  at  OSU.  Dr.  David  Bahner  will  aid  dissemination  efforts  to  emergency  and  trauma  care 
medicine  at  OSU,  Dr.  Michael  Barrie  will  facilitate  the  exploration  of  an  expansion  of  the  training  module 
for  fourth  year  medical  students  in  emergency  medicine,  and  Dr.  Susan  Moffatt-Bruce  will  serve  as  the 
interested  customer  for  purchasing  the  technology  and  associated  training.  We  will  develop  ‘Best 
practice’  principles  for  the  use  of  AR  with  simulation-based  training  for  medical  students  at  the  graduate 
and  undergraduate  level,  and  we  will  disseminate  these  principles  and  lessons  learned  via  professional 
conferences  in  medical  education. 

After  integrating  MART  into  the  curricula  at  Ohio  State  University,  we  will  be  better  positioned  to 
integrate  MART  into  the  TCCC  training  curricula.  We  plan  to  leverage  our  partnership  with  Innovative 
Tactical  Training  Solutions  (ITTS),  who  manufactures  the  TOMMankin.  ITTS  has  existing  agreements 
with  many  military  training  clients,  and  has  agreed  to  sell  the  pre-cursor  to  MART,  the  Virtual  Patient 
Immersive  Trainer  (VPIT).  Please  see  the  Commercial  Feasibility  section  in  the  conclusion  of  this  report 
for  more  details. 

Objective  3:  Create  Training  and  Evaluation  Scenarios 

Objective  3  was  to  create  two  realistic,  challenging  training  scenarios  —  one  for  tension  pneumothorax 
and  one  for  airway  obstruction.  We  also  created  expert  models  to  correspond  to  the  two  training 
scenarios.  Another  piece  of  Objective  3  was  to  create  evaluation  scenarios;  we  developed  a  framework  for 
creating  evaluation  scenarios  using  the  garden  path  scenario  method. 

Methods.  To  create  realistic,  engaging  scenarios,  we  interviewed  one  par arescue  jumper  (PJ)  and  three 
emergency  department  physicians.  We  used  Cognitive  Task  Analysis  (CTA)  interview  techniques 
(Militello  &  Hutton,  1998;  Hoffman  &  Militello,  2008)  to  elicit  a  case  base  of  real-life  incidents 
involving  tension  pneumothorax  and  airway  obstruction  injuries.  These  are  two  of  the  most  common 
treatable,  yet  potentially  fatal  injuries  experienced  on  the  battlefield  (Holcomb  et  al.,  2007;  Kelly  et  al., 
2008;  Gerhardt,  Mabry,  De  Lorenzo,  &  Butler,  2012).  CTA  techniques  are  used  to  articulate  the  cognitive 
aspects  of  a  particular  job,  especially  the  tacit  aspects  of  expertise.  CTA  methods  include  in-depth, 
incident-based  interviews  and  observations  of  expert  performance  in  naturalistic  or  simulated  settings. 

For  this  project,  we  employed  a  specific  CTA  method  called  the  Critical  Decision  Method  (CDM;  Klein, 
Calderwood,  &  MacGregor,  1989;  Crandall,  Klein,  &  Hoffman,  2006).  During  CDM  interviews, 
interviewers  guide  participants  through  recalling  an  incident  in  which  their  skills  were  challenged. 
Through  an  iterative  process,  the  interviewee  deepens  on  the  incident  in  terms  of  timeline,  critical  points 
during  the  incident,  goals,  optional  courses  of  action,  tradeoffs,  cue  utilization,  contextual  elements,  and 
situation  assessment  factors  associated  with  specific  decisions.  At  the  end  of  the  interview,  we  asked  the 
interviewee  a  series  of  hypothetical  questions,  including  where  a  less-experienced  person  might  have 
erred,  and  to  explore  the  implications  of  certain  cues  and  events.  During  these  discussions,  we  gathered 
stories  of  patients  who  suffered  from  tension  pneumothorax  and/or  airway  obstruction  injuries,  along  with 
elements  that  make  treating  these  injuries  difficult. 

Outcomes.  We  used  the  CTA  data  to  create  two  training  scenarios  (one  for  tension  pneumothorax,  one 
for  airway  obstruction).  All  four  subject  matter  experts  reviewed  draft  scenarios  for  realism,  and 
described  how  they  would  respond  at  each  decision  point.  These  responses  were  used  to  refine  the 
scenarios  and  build  an  expert  model  for  each.  Each  scenario  is  described  in  terms  of  point  of  injury 
context  and  key  signs  and  symptoms  at  set  time  points  during  treatment. 
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Training  Scenarios 

Scenario  1:  Tension  Pneumothorax.  An  explosive  ordnance  disposal  (EOD)  soldier  was  attempting  to 
neutralize  a  small  improvised  explosive  device  (IED).  The  IED  detonated  during  the  deactivation  attempt, 
causing  a  blast  lung  injury,  temporary  loss  of  consciousness,  blunt  force  trauma  to  the  rib  cage,  and  burns 
to  the  patient’s  hands.  The  patient  was  thrown  approximately  2  meters  away  from  the  blast  site,  and 
landed  on  pavement.  This  is  an  active  combat  situation,  so  a  key  concern  is  security.  Patient  is  a  male,  20- 
23  years  old,  approximately  180  lbs. 

•  Time  1:  30-60  seconds  after  point  of  injury  (POI):  At  this  point,  the  learner  is  able  to  do  a  quick 
assessment  of  the  patient,  but  needs  to  stay  alert  to  changing  combat  conditions.  The  patient  does 
not  show  signs  of  massive  hemorrhage,  but  there  is  some  blood  in  his  mouth.  The  patient  has 
close  to  full  respirations,  but  his  inhalation  volume  is  decreased.  He  has  a  fast  rate  of  breathing 
and  is  showing  some  signs  of  breathing  difficulty.  The  patient  is  alert,  but  is  in  pain  and  has 
superficial  and  partial  thickness  burns  to  his  hands.  Assessment  is  interrupted  by  active  fire. 

•  Time  2:  15  minutes  after  POI:  The  area  is  now  secure,  so  the  learner  is  able  to  continue  assessing 
the  patient.  The  learner  is  able  to  determine  that  there  are  broken  teeth  in  the  patient’s  mouth,  and 
it  looks  like  the  patient  bit  his  cheek  and  tongue  (explaining  the  blood  in  the  mouth  seen  at  Time 
1).  The  patient  has  bruises  on  his  right  rib  cage,  and  there  is  a  slight  paradoxical  motion  in  the  rise 
and  fall  of  his  chest.  The  patient  is  having  more  difficulty  and  increased  effort  of  breathing.  The 
patient  is  alert  and  talking  in  short  sentences.  The  patient  is  now  able  to  be  moved  to  a  more 
secure  area. 

•  Time  3:  45-60  minutes  after  POI:  The  patient  is  now  in  a  secure  and  protected  area,  and  the 
learner  is  able  to  re-assess.  Since  the  last  assessment,  the  patient  is  breathing  more  rapidly  and  is 
now  wheezing.  He  is  becoming  hypoxic  and  cyanotic.  The  paradoxical  motion  in  his  chest  is 
more  pronounced,  and  now  there  is  an  obvious  flail  chest  segment.  The  bruises  to  his  torso  are 
larger  and  have  changed  color.  The  learner  is  able  to  see  unilateral  rise  and  fall  of  his  chest.  The 
patient  is  confused  and  his  skin  is  pale. 

•  Time  4:  3  hours  after  POI:  The  patient  has  not  yet  reached  a  definitive  care  facility,  but  he  is 
sheltered  in  a  secure  location.  Upon  reassessment,  the  learner  notices  that  the  patient  is  coughing 
excessively  with  productive  red  sputum.  The  patient  has  decreased  breath  sounds  and  there  is 
blood  coming  out  of  his  mouth.  The  patient  is  showing  unilateral  rise  and  fall  of  the  chest,  and  his 
torso  bruising  has  worsened.  The  patient  shows  jugular  vein  distention  and  tracheal  deviation.  His 
skin  is  pale  and  he  is  in  and  out  of  consciousness. 

Scenario  2:  Airway  Obstruction.  A  rocket-propelled  grenade  (RPG)  struck  the  driver  side  front  door  of  an 
up-armored  vehicle  in  a  convoy.  The  driver  (wearing  a  ballistic  helmet,  body  armor,  and  gloves)  was 
closest  to  the  explosion  and  remained  entrapped  in  the  vehicle  for  approximately  90  seconds  before  his 
teammates  were  able  to  pull  him  from  the  burning  wreckage.  He  received  partial  and  full  thickness  burns 
around  his  neck,  jaw,  and  upper  cheeks.  The  body  armor  protected  his  chest  from  any  direct  burns  or 
penetrating  injury.  The  soldier’s  upper  extremities  were  hit  with  shrapnel  and  debris.  His  gloves  protected 
his  hands,  but  there  were  ring  burns  around  his  wrist. 

•  Time  1:  Immediately  following  POI:  The  learner  is  able  to  get  to  the  patient  shortly  after  he  was 
removed  from  the  burning  vehicle.  After  getting  the  patient  to  a  secure  location,  the  learner 
assesses  the  patient.  While  there  is  blood  on  his  uniform,  it  is  from  lacerations  from  debris;  there 
is  no  massive  hemorrhage.  The  patient  has  singed  nostril  hairs  and  the  skin  on  his  face  and  his 
lips  is  turning  red,  like  he  has  been  sunburned.  There  are  also  small  pieces  of  shrapnel  in  his  face. 
There  are  no  major  secretions  in  his  mouth  and  his  neck  looks  normal.  His  voice  is  hoarse  and  he 
is  coughing.  The  patient  has  increased  respirations,  but  his  chest  movement  is  normal.  Patient  is 
conscious  but  in  pain  and  seems  to  be  rambling.  He  is  sweating  and  has  light  burns  on  his  arms. 

•  Time  2:  5-10  minutes  after  POI:  The  patient  has  become  lethargic  and  is  in  obvious  respiratory 
distress,  so  you  reassess.  You  notice  that  the  tissue  along  his  airway  is  beginning  to  swell,  and  the 
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patient  is  coughing  up  thick  carbonaceous  sputum  with  some  yellow-green  phlegm  and  blood. 

The  skin  around  the  patient’s  mouth  is  becoming  a  darker  red,  and  blisters  are  forming  across  his 
cheeks  and  forehead.  The  patient’s  nostrils  are  becoming  smaller,  and  there  is  general  swelling 
around  his  face.  There  are  more  secretions  in  his  mouth,  and  his  pharynx  is  starting  to  close  up. 

•  Time  3:15  minutes  after  POI:  Non-surgical  attempts  at  managing  the  patient’s  airway  have 
failed,  and  the  patient’s  condition  is  worsening.  The  tissue  in  his  nose  and  mouth  is  significantly 
swollen;  his  lips  are  triple  in  size.  Larger  blisters  are  starting  to  form  on  his  face.  The 
carbonaceous  sputum  and  secretions  in  the  mouth  are  very  prominent.  The  learner  hears  stridor, 
wheezing,  and  wet  lung  sounds.  The  patient  has  an  increased  effort  and  rate  of  breathing.  The 
patient  is  fatigued  and  has  a  muted  cough. 

Each  of  these  scenarios  describe  the  trajectory  of  the  patients’  injuries  if  appropriate  medical  interventions 
are  not  applied.  The  virtual  patients  were  developed  to  reflect  these  scenarios. 

Expert  Models 

We  aggregated  SME  responses  to  the  cognitive  probes  associated  with  each  training  scenario.  We  then 
coded  responses  and  identified  overlapping  content  areas.  From  these,  we  created  expert  models  for  each 
scenario,  representing  which  cues  the  experts  found  most  important  at  each  time  point,  their  differential 
diagnoses,  planned  actions,  and  expected  prognosis  for  the  patient.  Team  member  Oliver  Smith  (PJ, 

CRO),  reviewed  and  refined  the  integrated  expert  models  to  ensure  that  they  represented  a  combat  medic 
perspective.  Full  scenarios  with  the  associated  expert  models  are  included  in  Appendix  A. 

Evaluation  Scenarios 

We  tailored  the  Garden  Path  scenario  approach  to  develop  evaluation  scenarios.  Our  original  intent  was  to 
create  two  Garden  Path  scenarios  to  be  used  in  a  summative  evaluation.  Elowever,  we  have  modified  our 
evaluation  approach  to  include  both  formative  and  summative  testing.  To  accommodate  this  shift,  we 
created  a  framework  that  will  allow  us  to  quickly  create  Garden  Path  evaluation  scenarios  to  incorporate 
into  any  type  of  evaluation  (see  Figure  1 1  below). 

Garden  Path  scenarios  (Patterson,  Roth,  &  Woods,  2010)  are  useful  for  testing  macrocognitive  skills, 
such  as  sensemaking  and  problem  detection.  According  to  Patterson,  Miller,  Roth,  and  Woods  (2010), 
sensemaking  consists  of  collecting  information  and  assessing  how  it  fits  potential  explanations.  Problem 
detection  is  noticing  whether  events  are  unfolding  as  expected  or  not,  and  whether  a  reframing  of  the 
situation  is  needed.  Garden  path  scenarios  are  designed  to  assess  how  well  and  how  quickly  participants 
detect  and  synthesize  information  into  correct  hypotheses. 

A  garden  path  scenario  begins  with  cues  and  signals  pointing  very  clearly  to  a  conceptualization  of  the 
situation  (i.e.,  diagnosis)  that  appears  accurate,  but  is  ultimately  incorrect.  This  original  answer  is  known 
as  th e  false  prime  hypothesis.  When  used  in  nuclear  power,  this  false  prime  hypothesis  could  be  the  result 
of  a  reading  from  a  faulty  sensor  that  the  operator  does  not  realize  is  malfunctioning.  The  false  prime  is 
the  result  of  bad  information  and  leads  the  participant  down  the  garden  path. 

As  the  garden  path  scenario  unfolds,  additional  cues  and  signals  are  presented  to  the  participant  as  either 
injects  or  discoverables.  Injects  are  changes  to  the  scenario  that  are  unprompted  by  the  participant  (e.g., 
“The  patient  is  now  unconscious”).  Discoverables  are  cues  that  are  available  upon  request  by  the 
participant  (e.g.,  the  participant  asks  the  experimenter,  “What  is  the  patient’s  heart  rate?”,  and  the 
experimenter  gives  the  answer).  Discoverables  are  akin  to  looking  at  instruments  or  test  results,  but 
require  information-seeking  behaviors  on  the  part  of  the  participant.  These  additional  cues  and  signals  are 
presented  as  subtle  changes  that  indicate  the  underlying  true  explanation  for  what  is  occurring  in  the 
scenario.  If  the  participant  detects  these  changes  and  realizes  that  they  do  not  fit  into  the  initial  false  prime 
hypothesis,  s/he  should  start  to  develop  alternative  hypotheses. 

The  main  unit  of  measurement  in  a  garden  path  scenario  is  time.  We  can  measure  how  long  it  takes 
participants  to  discard  the  initial  false  prime  hypothesis,  how  long  it  takes  participants  to  generate 
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alternative  hypotheses,  and  how  long  it  takes  them  to  arrive  at  the  correct  explanation  for  what  is 
occurring  in  the  scenario.  De  Keyser  and  Woods  (1990)  found  that  participants  held  on  to  the  original 
hypothesis  (that  was  ultimately  incorrect)  too  long,  failing  to  adjust  their  thinking  during  dynamic 
diagnosis  in  a  complex  setting.  However,  Hoffman  and  Fiore  (2007)  found  that  experts  are  less  likely  to 
be  trapped  in  a  garden  path  scenario  and  are  better  at  recognizing  the  subtle  perceptual  cues  that  lead  to 
the  correct  outcome.  Novices  tend  to  hold  on  to  the  initial  hypothesis  longer  than  experts,  who  are  better 
able  to  perceive  and  interpret  the  subtle  cues  that  indicate  a  different  hypothesis  would  be  more 
appropriate. 

While  garden  path  scenarios  have  been  applied  to  military  and  nuclear  power  domains,  they  are  less 
commonly  used  in  healthcare  research.  We  modified  the  traditional  garden  path  approach  in  a  few  ways 
to  better  fit  the  domain  of  emergency/combat  medicine.  First,  traditionally  the  false  prime  hypothesis  is 
completely  wrong.  However,  we  recognize  that  trauma  patients  often  have  multiple  injuries,  some 
obvious  and  some  obscure.  The  SMEs  that  we  interviewed  said  that  the  most  challenging  cases  are  when 
there  are  distracting  injuries  that  appear  bad,  but  are  masking  a  more  insidious  condition  that  will  actually 
kill  the  patient  if  left  unaddressed.  Therefore  our  garden  path  scenarios  will  contain  a  false  prime 
hypothesis  that  is  related  to  the  complete  diagnosis,  but  is  the  wrong  injury  to  focus  on.  For  example,  a 
partial  limb  amputation  could  be  the  false  prime  hiding  an  airway  obstruction.  The  partial  limb 
amputation  is  serious  and  might  kill  the  patient,  but  the  patient  will  definitely  die  if  the  airway  is  not 
secured.  Because  it  is  rare  that  a  trauma  patient  would  only  have  one  or  two  injuries,  our  garden  path 
scenarios  will  have  a  false  prime  hypothesis,  several  alternative  hypotheses,  and  a  complete  diagnosis 
which  will  be  a  combination  of  the  previously-generated  hypotheses. 

We  will  measure  the  time  that  it  takes  participants  to  rule  out  the  false  prime  hypothesis  as  the  primary 
explanation  for  what  is  wrong  with  the  patient,  instead  of  ruling  the  false  prime  out  completely.  We  will 
also  measure  how  long  it  takes  participants  to  generate  correct  alternative  hypotheses,  and  how  long  it 
takes  them  to  arrive  at  the  complete  diagnosis.  At  the  end  of  the  scenario,  we  will  reveal  the  correct 
diagnosis  to  support  reflection  and  after-action  review. 
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Figure  11.  Visual  depiction  of  a  basic  garden  path  scenario  comparing  performance  with  two  conditions 
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Objective  4:  Design  an  Evaluation  to  Test  AR  Training  Technology 

We  developed  a  plan  for  a  series  of  formative  evaluations  and  user  feedback  sessions  to  test  the 
effectiveness  of  specific  components  of  MART  to  be  carried  out  in  Phase  II.  The  formative  evaluations 
will  be  a  series  of  small-n  controlled  studies  comparing  an  experimental  group  exposed  to  a  specific 
MART  adaptive  training  component,  and  a  control  group  that  is  not.  Focused  user  feedback  sessions  will 
include  one-on-one  sessions  with  combat  medics,  medical  students,  and  instructors  to  explore  issues  of 
usability,  perceived  benefits  and  drawbacks,  and  strategies  for  integrating  into  existing  training.  Results 
from  the  formative  studies  and  user  feedback  sessions  will  be  directly  incorporated  into  our  software 
development  cycle.  We  also  planned  for  a  summative  evaluation  study  near  the  end  of  Phase  II  to  test  a 
complete  MART  training  module.  The  team  from  our  university  partner  (Ohio  State  University),  led  by 
Dr.  Emily  Patterson,  worked  on  creating  materials  and  articulating  the  method  for  the  summative 
evaluation.  The  outcomes  of  this  series  of  evaluations  will  be  a  set  of  AR  training  design  principles  that 
can  be  generalized  to  other  AR  training  applications. 

Method.  We  held  several  meetings  with  the  project  team  to  articulate  important  aspects  of  AR  that  we 
believe  will  impact  training.  We  also  reviewed  literature  related  to  the  use  of  simulations,  AR,  and  virtual 
reality  (VR)  in  medical  training.  We  articulated  a  series  of  research  questions  based  on  these  AR 
elements,  and  chose  the  ones  that  seemed  the  most  impactful.  The  goal  of  the  evaluation  effort  is  to 
determine  if  the  MART  system  improves  training  along  with  how  the  system  improves  training. 

Outcomes.  The  evaluation  plan  leverages  both  formative  and  summative  evaluation  activities  to  test 
components  of  the  MART  system,  and  inform  evidence-based  principles  for  designing  augmented  reality 
training.  We  will  conduct  a  summative  evaluation  near  the  end  of  the  Phase  II  in  which  we  will  compare 
the  MART  system  to  existing  training.  Table  1  summarizes  the  evaluation  efforts  in  terms  of  the  training 
design  principle  (left  column),  evaluation  question  (middle  column),  and  performance  measures  (right 
column).  Details  for  the  formative  and  summative  evaluations  are  provided  below  the  table. 

Table  1 .  Phase  II  evaluation  plan 


AR  Training  Design  Principle 

Guiding  Question 

Measures 

Formative  Studies 

1 .  Embed  AR-generated  realistic 
cues,  including  multi -dimensional, 
multi-sensed,  dynamic  cues  to 
support  perceptual  skill 
development. 

Do  realistic  cues  facilitate  perceptual 
skill  development  better  than 
concept  drawings? 

Ability  to  distinguish  healthy  from 
injured: 

•  Rise  and  fall  of  the  chest 

•  Breath  sounds 

•  Airway 

2.  Embed  attention-directing  hints 

from  an  AR-generated  mentor  avatar 
following  incomplete  assessment  to 
support  cue  detection. 

Do  attention-directing  hints  from  an 
avatar  improve  the  learner’s  ability 
to  accurately  and  efficiently  assess  a 
patient’s  condition? 

Ability  to: 

•  Recognize  critical  cues 

•  Recognize  critical  cues  quickly 

•  Conduct  a  thorough  assessment 

•  Match  to  expert 

3.  Embed  real-time  auditory 
corrections  following  erroneous 
actions  to  provide  immediate 
feedback. 

Do  real-time  auditory  corrections 
improve  learners’  ability  to 
accurately  and  efficiently  assess  a 
patient’s  condition? 

Ability  to: 

•  Recognize  critical  cues 

•  Recognize  critical  cues  quickly 

•  Conduct  a  thorough  assessment 

•  Match  to  expert 

4.  Augment  the  external  view  of  the 
patient  with  animations  of 
underlying  dynamic  anatomy  to 
support  mental  model  development 
of  injury  progression,  including 
anticipatory  reasoning. 

Does  exposure  to  depictions  of 
underlying  dynamic  anatomy  in  the 
context  of  a  challenging  scenario 
improve  understanding  of  injury  and 
ability  to  predict  likely  outcomes? 

Ability  to: 

•  Accurately  diagnose  injury 

•  Quickly  diagnose  injury 

•  Predict  likely  outcomes  with  no 
intervention 

•  Predict  likely  outcomes  with 
appropriate  intervention 
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AR  Training  Design  Principle 

Guiding  Question 

Measures 

User  Feedback  Sessions 

5.  Include  time  manipulation  to 

support  development  of  mental 
models  of  injury  progression 
including  anticipatory  reasoning. 

Do  instructors  and  students 
anticipate  that  “fast  forwarding”  in 
time  to  see  how  an  injury  progresses 
without  treatment  improves 
understanding  of  injury  and  ability 
to  predict  likely  outcomes? 

6.  Support  self-reflective  learning 
using  an  AR-generated  mentor  that 
encourages  learners  to  articulate 
assessments,  hypotheses,  and 
rationale  to  facilitate  mental  model 
development. 

Do  instructors  and  students 
anticipate  that  articulating 
assessments  and  hypotheses  will  aid 
learners  in  developing  mental 
models  to  support  assessment  skills? 

7.  Strategies  to  encourage  reflection 
and  feedback  during  AR  training 
can  improve  learning  outcomes. 

Can  the  proposed  reflection  and 
feedback  components  including 
comparison  to  expert  model,  replay, 
ability,  and  performance  scores,  be 
effectively  integrated  into  after¬ 
action  review? 

Summative  Study 

8.  AR  adaptive  training  has 

advantages  over  existing  training. 

Is  AR  adaptive  training  more 
effective  than  existing  training? 

See  Table  2  in  next  section 

Institutional  Review  Board  (IRB)  approval  for  research  activities  will  be  requested  from  OSU  in  a  two- 
stage  approach,  with  formative  evaluations  submitted  first  to  avoid  delays  if  selected  for  funding.  In  the 
second  stage,  the  validation  study  will  be  submitted  as  an  amendment  once  the  exact  technology  and 
training  to  be  validated  has  been  finalized.  The  first  human  subjects  research  activities  are  planned  for  the 
end  of  the  second  quarter  of  the  Phase  II  project.  Dr.  Patterson  (Co-PI  at  OSU)  turned  in  the  just-in-time 
IRB  submission  to  the  OSU  IRB  in  early  April  2018. 

Formative  Studies 

Formative  studies  will  be  conducted  with  medical  students  at  the  OSU  College  of  Medicine.  Each  will 
include  10  students  in  an  experimental  group  who  receive  a  key  training  component,  and  10  in  a  control 
group  who  do  not. 

Study  1:  Do  realistic  cues  facilitate  perceptual  skill  development  better  than  concept  drawings? 
Participants:  First  and  second  year  medical  students. 

Materials:  For  Study  1,  we  will  develop  a  training  module  that  includes  perceptual  cues  depicting  healthy 
and  injured  anatomy  on  virtual  patients  for  the  experimental  condition.  We  will  build  an  analogous 
training  module  for  the  control  group  leveraging  descriptions,  photos,  drawings,  and/or  animations  used 
in  existing  training  programs.  Candidate  perceptual  skills  might  include  distinguishing: 

•  Healthy  versus  atypical  breath  sounds 

•  Healthy  versus  asymmetrical  rise  and  fall  of  the  chest 

•  Healthy  versus  jugular  vein  distention 

•  Healthy  versus  atypical  skin  tone 

•  Healthy  versus  swollen  airway 

•  Properly  placed  versus  kinked  needle  decompression 

•  Properly  placed  versus  leaky  occlusive  dressing 

We  will  develop  a  test  to  assess  learners'  ability  to  make  the  perceptual  distinctions  described  above. 
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Procedure:  After  informed  consent,  participants  will  be  asked  to  complete  a  pre-test  assessing  their 
ability  to  distinguish  important  perceptual  cues.  Each  participant  will  work  through  either  the 
experimental  or  control  training  module.  After  finishing  the  training  module,  participants  will  complete  a 
post-test  consisting  of  the  same  questions  used  in  the  pre-test. 

We  will  compare  changes  in  time  and  accuracy  from  pre-  to  post-test  between  the  experimental  and 
control  groups  to  determine  whether  the  use  of  realistic  cues  facilitate  perceptual  skill  development. 

Study  2:  Do  attention-directing  hints  from  an  avatar  improve  the  learner’s  ability  to  accurately  and 

efficiently  assess  a  patient’s  condition? 

Participants:  Second  and  third  year  medical  students 

Materials:  For  Study  2,  we  will  develop  an  AR-generated  avatar  for  use  by  the  experimental  group.  The 
avatar  will  appear  as  the  voice  of  a  more  experienced  medic  (or  physician)  encouraging  the  learner  to 
notice  or  look  for  specific  cues.  We  will  develop  pre-  and  post-test  scenarios  of  similar  difficulty 
depicting  similar  cues  and  conditions. 

Procedure:  After  informed  consent,  participants  will  work  through  a  pre-test  scenario.  They  will  be 
prompted  to  describe  what  they  are  noticing,  an  assessment  of  the  patient,  and  recommended 
interventions.  Gaze  tracking  software  will  record  when  they  look  at  specific  cues.  The  experimental  group 
will  work  through  two  training  scenarios  using  the  AR-generated  avatar  providing  attention-directing 
hints.  The  control  group  will  work  through  the  same  two  training  scenarios  without  attention-directing 
hints.  Both  groups  will  complete  the  post-test  scenario,  again  articulating  what  they  are  noticing,  an 
assessment  of  the  patient,  and  recommended  interventions. 

We  will  compare  changes  from  Time  1  to  Time  2  in  learners’  ability  to  notice  critical  cues,  quickly  and 
accurately  assess  the  patient,  and  identify  an  appropriate  course  of  action.  Performance  will  be  scored 
against  an  expert  model. 

Study  3:  Do  real-time  auditory  corrections  improve  learners  ’  ability  to  accurately  and  efficiently  assess  a 

patient’s  condition? 

Participants:  First  year  medical  students 

Materials:  For  Study  3,  we  will  develop  an  auditory  correction  that  appears  when  the  learner  deviates 
from  the  ABCDEF  assessment  algorithm  to  be  used  by  the  experimental  group.  We  will  use  the  pre-  and 
post-test  scenarios  developed  for  Study  2. 

Procedure:  After  informed  consent,  participants  will  work  through  a  pre-test  scenario.  They  will  be 
prompted  to  describe  what  they  are  noticing  as  they  assess  the  patient.  Gaze  tracking  software  will  record 
when  they  look  at  specific  cues.  The  experimental  group  will  work  through  two  training  scenarios  using 
the  real-time  auditory  corrections.  The  control  group  will  work  through  the  same  two  training  scenarios 
without  correction.  Both  groups  will  complete  the  post-test  scenario,  again  articulating  what  they  are 
noticing  as  they  assess  the  patient. 

We  will  compare  changes  from  Time  1  to  Time  2  in  learners’  ability  to  work  through  the  assessment 
algorithm  in  the  correct  order. 

Study  4:  Does  exposure  to  depictions  of  underlying  dynamic  anatomy  in  the  context  of  a  challenging 
scenario  improve  understanding  of  injury  and  ability  to  predict  likely  outcomes? 

Participants:  First  year  medical  students. 

Materials:  For  Study  4,  we  will  develop  animations  that  show  underlying  dynamic  anatomy  in  the 
context  of  tension  pneumothorax  and  airway  obstruction  for  use  by  the  experimental  group.  We  will  use 
the  pre-  and  post-test  scenarios  developed  for  Study  2. 
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Procedure:  After  informed  consent,  participants  will  work  through  a  pre-test  scenario.  They  will  be 
prompted  to  describe  what  they  are  noticing,  an  assessment  of  the  patient,  and  recommended 
interventions.  They  will  be  asked  to  describe  how  the  patient’s  condition  will  progress  if  untreated, 
including  specific  prompts  about  the  underlying  dynamic  anatomy  (i.e.,  what  is  happening  in  the  patient’s 
chest  cavity).  Gaze  fracking  software  will  record  when  they  look  at  specific  cues.  The  experimental  group 
will  work  through  two  training  scenarios  using  the  dynamic  anatomy  animations.  The  control  group  will 
work  through  the  same  two  training  scenarios  without  dynamic  anatomy  animations.  Both  groups  will 
complete  the  post-test  scenario,  responding  to  the  same  probes  used  in  the  pre-test. 

We  will  compare  changes  from  Time  1  to  Time  2  in  learners’  ability  to  notice  critical  cues,  quickly  and 
accurately  assess  the  patient,  identify  an  appropriate  course  of  action,  and  describe  the  underlying 
dynamic  anatomy. 

User  Feedback  Sessions 

User  feedback  sessions  will  be  used  to  obtain  feedback  about  additional  design  and  framing  components. 
Sessions  will  include  a  combination  of  medical  students,  combat  medics,  and/or  instructors.  We  plan  to 
recruit  combat  medics  from  the  256th  Combat  Hospital  U.S.  Army  Reserve  Unit,  and  from  the  student 
population  at  OSU.  We  will  continue  pursuing  contacts  with  combat  medic  training  organizations  (i.e.. 
Military  Education  and  Training  Command).  User  feedback  sessions  may  occur  at  early  design  stages, 
working  with  drawings  and  design  concepts,  and  later  as  more  detailed  prototypes  become  available. 
Sessions  that  take  place  earlier  in  the  design  and  development  cycle  will  have  smaller  samples  (i.e.,  1-5 
participants);  those  that  take  place  later  may  have  up  to  10  participants.  In  these  sessions,  participants  will 
be  asked  to  envision  using  the  proposed  feature  in  the  context  of  real-world  training.  We  will  explore 
perceived  benefits  and  drawbacks,  feedback  about  the  usability  of  the  proposed  design,  and  anticipated 
barriers  to  integration  into  training.  When  we  have  detailed  prototypes,  participants  will  interact  with  the 
prototype  and  complete  a  usability  survey  (i.e.,  System  Usability  Scale,  Lewis  &  Sauro,  2009;  Sauro, 

201 1).  We  anticipate  that  topics  for  user  feedback  will  emerge  during  the  design  and  development 
process;  however,  we  have  identified  three  topics  for  user  feedback  at  this  early  stage. 

Do  instructors  and  students  anticipate  that  “fast  forwarding’’  in  time  to  see  how  an  injury  progresses 
without  treatment  improves  understanding  of  the  injury  and  ability  to  predict  likely  outcomes? 

One  topic  for  user  feedback  sessions  is  time  manipulation.  Although  the  notion  is  intuitively  appealing, 
time  manipulation  can  be  used  in  several  ways.  It  can  be  used  to  extend  a  scenario,  to  depict  outcomes  as 
part  of  an  after-action  review,  or  as  paid  of  a  tutorial.  User  feedback  sessions  with  instructors  and  learners 
will  help  us  narrow  our  focus  to  identify  the  most  promising  use  of  time  manipulation,  and  to  answer  the 
question  of  whether  instructors  and  students  anticipate  that  this  manipulation  will  improve  training. 

Do  instructors  and  students  anticipate  that  articulating  assessments  and  hypotheses  will  aid  learners  in 
developing  mental  models  to  support  assessment  skills? 

Another  topic  for  which  we  will  obtain  user  feedback  is  related  to  the  research  literature  that  suggests  that 
asking  learners  to  articulate  assessments,  hypotheses,  and  rationale  as  they  work  may  support  mental 
model  development  (Klein  &  Wolf,  1995;  Patterson,  Militello,  Bunger,  Taylor,  Klein,  &  Geis,  2016). 
Anecdotal  advice  from  combat  medic  instructors,  however,  indicates  that  encouraging  this  during  training 
may  be  too  much  of  a  deviation  from  recommended  practice  (e.g.,  medics  do  not  announce  their  thoughts 
as  they  work  on  patients  in  the  real  world).  User  feedback  sessions  will  allow  us  to  explore  different 
strategies  for  supporting  mental  model  development  in  this  way  before  we  embark  on  detailed  design  for 
a  prototype. 

Can  the  proposed  reflection  and  feedback  components  including  comparison  to  expert  model,  replay 
ability ,  and  performance  scores  be  effectively  integrated  into  after-action  review? 

A  third  topic  is  that  of  encouraging  reflection  and  feedback.  The  proposed  performance  summary  is 


22 


intended  to  serve  this  purpose.  To  be  effective  it  must  be  compatible  with  existing  after-action  review  and 
other  feedback  strategies.  We  will  obtain  user  feedback  from  instructors  and  learners  to  better  understand 
existing  constraints  and  how  we  can  leverage  AR  adaptive  training  to  support  both  the  learner  and  trainer 
in  reflection  and  feedback. 

Summative  Evaluation 

Findings  from  formative  studies  and  user  feedback  sessions  will  guide  the  final  design  of  the  MART 
prototype  to  be  assessed  in  the  summative  evaluation.  Our  primary  hypothesis  for  the  summative 
evaluation  is  that  AR-based  adaptive  training  as  instantiated  in  MART  will  enable  more  timely  diagnosis 
and  treatment  as  compared  to  ‘textbook’  written  and  verbal  descriptions  of  cues  (e.g.,  ‘increasing 
respiratory  distress’).  We  will  explore  the  question  of  whether  MART  improves  perceptual  cue 
recognition,  sensemaking,  assessment,  treatment  planning  skills,  and  mental  model  development.  We  will 
assess  student  reactions  and  learning  of  foundational  knowledge. 

Participants:  For  the  summative  evaluation,  we  will  recruit  40  participants.  Participants  will  be  OSU 
medical  students.  Participants  will  be  reimbursed  for  participation  at  $40/hour. 

Materials:  We  will  evaluate  a  version  of  the  MART  system  that  is  specific  to  emergency  department 
training  ( MART-ED )  tailored  for  use  by  the  OSU  College  of  Medicine.  This  version  will  incorporate 
adaptive  training  elements  deemed  most  useful  based  on  formative  studies  and  user  feedback  sessions.  In 
order  to  reduce  asymmetric  learning  effects  associated  with  using  the  FIoloLens,  study  participants  will 
use  a  tablet-based  AR  training  application  rather  than  the  FIoloLens. 

Training  and  evaluation  scenarios  will  be  nearly  identical  conceptually,  with  changes  limited  to 
demographic  variables  (e.g.,  58  years  old  instead  of  45  years  old)  and  medical  conditions  which  do  not 
directly  interact  (e.g.,  other  co-occurring  injuries  to  other  body  parts  such  as  a  broken  arm  vs.  a  broken 
leg).  In  evaluation  scenarios,  unlike  the  training  scenarios,  there  will  be  no  requirements  to  stop  treating 
the  patient  in  order  to  deal  with  distracting  events  in  the  environment  or  other  patients  in  order  to  ensure 
the  integrity  of  the  time  measure.  The  evaluation  scenarios  will  be  designed  to  be  Garden  Path  scenarios. 

Design:  The  study  will  include  a  pre-post,  between  subjects  design  where  half  of  the  participants  receive 
traditional  classroom  training  and  half  of  the  participants  receive  the  intervention  of  the  AR  training. 
Participants  are  randomized  to  baseline  (traditional  classroom  training)  as  compared  to  intervention 
condition  (AR  training). 

Procedure:  After  obtaining  informed  consent,  participants  will  be  assigned  to  their  condition  and  take  the 
pre-test  battery.  They  will  receive  training  (either  AR  or  conventional),  then  complete  the  post-test 
battery,  including  the  Garden  Path  evaluation  scenarios.  The  Garden  Path  evaluation  scenarios  will  be 
video-recorded.  Sessions  will  last  2  hours. 

Measures:  For  this  study,  we  will  integrate  macrocognitive  measures  with  more  traditional  instructional 
system  design  (ISD)  measures.  These  two  perspectives  are  complementary  and  overlap  in  important 
ways.  Table  3  provides  an  overview  of  measures  to  be  used  in  the  summative  evaluation.  Note:  ISD 
constructs  are  described  in  terms  of  those  articulated  by  Kirkpatrick  (1998)  and  Miller  (1990)  in  their 
discussion  regarding  what  constitutes  meaningful  evaluation  of  training  interventions. 

Table  2.  Overview  of  summative  evaluation  measures 


Measure 

Macrocognition 

ISD 

Data 

1.  Time  to  rule  out  false  prime 

Sensemaking 

Time 

2.  Time  to  full  diagnosis 

Sensemaking 

Time 

3.  Proportion  of  critical  cues 
identified 

Perceptual  skills 

Think  aloud  statements 

4.  Proportion  of  cues  scanned 

Assessment  skills 

Gaze  tracking 

5.  Order  cues  are  scanned 

Assessment  skills 

Gaze  tracking 
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Measure 

Macrocognition 

ISD 

Data 

6.  Alternative  explanations 
considered 

Sensemaking 

Think  aloud  statements 

7.  Action  priorities 

Treatment  planning 

Video  record/observation 

8.  Link  dx  to  treatment  priorities 

Mental  models 

Rationale  statements 

9.  Knowledge  of  anatomy 

Mental  models 

Learning 

Knowledge  test 

10.  Knowledge  of  treatment 
priorities 

Mental  models 

Learning 

Knowledge  test 

11.  Usability  and  usefulness 

Reaction 

Survey 

During  analysis,  the  collected  video  data  will  be  compared  to  an  expert  model  of  performance  using  a 
theoretical  framework  of  macrocognitive  measures.  Specifically, 

•  The  times  needed  to  rule  out  the  false  prime  explanation  (Measure  1)  and  to  come  to  an  accurate 
diagnosis  (Measure  2)  will  be  compared  with  expectations  for  experts; 

•  The  cues  identified  as  critical  by  participants  (Measure  3)  will  be  compared  to  critical  cues 
identified  by  experts; 

•  The  proportion  of  possible  cues  to  detect  (Measure  4)  and  the  order  in  which  they  are  scanned 
(Measure  5)  to  aid  in  diagnosis/sensemaking  will  be  compared  to  the  full  possible  set  that  experts 
would  detect  based  upon  gaze  tracking  data; 

•  Think-aloud  statements  (Measure  6)  will  provide  evidence  that  participants  considered  alternative 
explanations  (differential  diagnosis)  or  the  possibility  of  masked  conditions  (interacting 
diagnoses),  which  is  a  marker  of  high  expertise  in  sensemaking; 

•  Actions  taken  to  ‘safe’  a  patient  such  as  stabilizing  blood  pressure  or  ensuring  an  intact  airway 
(Measure  7)  will  be  used  as  evidence  for  expert  management  of  the  trade-offs  between  diagnosis 
and  stabilization  of  the  patient;  and 

•  The  clarity  of  justifications  for  inferences  relating  to  diagnosis  and  how  to  prioritize  treatment 
actions  (Measure  8)  will  be  assessed  by  emergency  medicine  SMEs  in  relation  to  level  of 
sophistication  and  clarity. 

It  is  important  to  note  that  these  are  scenario-specific  measures.  For  example,  a  directly  observable 
performance  measure  for  the  tension  pneumothorax  scenario  is  the  primary  measure  of  time  until  full 
diagnosis  of  a  blast  injury  resulting  in  a  tension  pneumothorax.  The  Garden  Path  scenario  will  point  to 
a  different  diagnosis;  in  this  way,  there  will  be  an  objective  ‘start’  time  for  measurement  until  the 
participant  has  reached  an  accurate  diagnosis.  For  the  airway  obstruction  scenario,  the  primary  measure 
is  time  until  diagnosis  of  a  burn  injury  as  a  result  of  breathing  superheated  air,  resulting  in  airway 
obstruction. 

In  addition  to  macrocognitive  measures,  we  will  implement  measures  recommended  by  Kirkpatrick 
(1998)  and  Miller  (1990).  When  applying  this  evaluation  framework,  the  focus  will  include  survey 
responses  regarding  attitudes  about  the  training  itself  (Level  1  evaluation),  test  scores  assessing 
knowledge  (Level  2  evaluation),  and  directly  observable  performance  measures  (Level  3  evaluation). 
These  ISD-based  components  establish  a  critical  baseline  for  evaluation.  Specifically,  the  participants 
will  complete: 

•  A  knowledge  test  regarding  relevant  anatomy  (Measure  9), 

•  A  knowledge  test  regarding  which  treatment  activities  to  prioritize  with  respect  to  the  scenario- 
specific  elements  (e.g.,  ensure  there  is  no  internal  bleeding  before  assessing  the  extent  of  burn 
injuries  on  skin;  Measure  10),  and 

•  A  usability  and  usefulness  survey  (Measure  1 1).  The  usefulness  survey  will  aid  in  assessing 
whether  participants  judge  that  the  virtual  patient  reacts  as  a  human  would  in  the  real  world, 
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whether  they  find  the  training  experience  valuable,  and  will  include  comparison  to  alternative 
platforms  such  as  online  Paining  delivery. 

Conclusions.  This  series  of  evaluations  will  test  the  effectiveness  of  MART,  along  with  articulating  how 
the  system’s  components  affect  training.  Results  from  the  formative  evaluations  will  be  integrated  into  the 
software  development,  in  line  with  user-centered  design  principles.  We  will  use  medical  school  students 
and  resident  physicians  for  most  of  the  studies  because  they  are  more  accessible  than  combat  medics. 
However,  we  will  incorporate  combat  medic  feedback  through  their  involvement  in  the  formative  user- 
feedback  sessions. 

The  use  of  augmented  and  virtual  reality  medical  training  has  proliferated  in  recent  years.  Although  these 
efforts  are  often  met  with  enthusiasm,  limited  research  has  been  conducted  to  identify  effective  strategics 
for  designing  and  implementing  AR-based  medical  Paining,  resulting  in  a  broad  range  of  applications 
with  little  understanding  of  strengths  and  weaknesses  of  different  approaches  and  techniques  (Barsom, 
Graafland,  &  Schijven,  2016;  Zhu,  Hadadgar,  Masiello,  &  Zary,  2014).  Research-based  design  principles 
are  needed  to  articulate  best  practices  in  AR-based  training  design.  The  outcome  of  this  series  of  studies 
will  be  a  better  understanding  of  how  specific  AR  and  adaptive  Paining  elements  affect  training  outcomes 
in  the  medical  domain. 

3.  KEY  RESEARCH  ACCOMPLISHMENTS 

The  key  research  accomplishments  of  this  Phase  I  project  include: 

1 .  Successful  demonsPations  of  novel  solutions  to  virtual  patient  registration  to  a  physical  manikin. 

2.  Created  challenging,  realistic  training  scenarios  with  associated  expert  models. 

3.  Adapted  the  garden  path  scenario  method  for  use  in  emergency  medicine  domain. 

4.  Built  virtual  patient  assets  that  could  be  incorporated  into  existing  AR  software  platforms. 

5.  Identified  adaptive  training  components  that  AR  affords  during  the  completion  of  realistic 
scenarios. 

6.  Developed  a  sPategy  for  evaluating  the  effectiveness  of  individual  Paining  components  that  will 
lead  to  theoretically-based  principles  for  designing  AR  adaptive  training. 

4.  CONCLUSION 

Augmented  reality  has  the  potential  to  improve  on  the  already  successful  TCCC  training.  The  ability  to 
present  realistic  visual  and  auditory  cues  allows  the  learner  to  practice  perceptual  skills  in  a  simulated 
environment  as  never  before.  By  integrating  the  AR  virtual  patient  with  a  physical  manikin  and 
embedding  the  Paining  in  compelling  realistic  scenarios,  we  can  create  a  holistic  training  experience  that 
exercises  perceptual  skills,  assessment,  sensemaking,  and  the  application  of  medical  procedures  in  a 
unified  training  experience.  Furthermore,  AR  opens  the  door  for  including  a  range  of  adaptive  training 
features  in  the  simulated  Paining  experience.  The  promise  of  a  sea  change  in  combat  medic  Paining  is 
inviting;  however,  it  is  important  to  document  lessons  learned  throughout  the  design  process,  and  to 
assess  the  effectiveness  of  individual  components  of  AR-based  adaptive  training.  The  goals  of  this 
research  program  are  two-fold:  1)  to  improve  combat  medic  Paining  and  2)  to  develop  a  set  of  evidence- 
based  principles  to  guide  the  design  of  AR  training. 

This  Phase  I  project  moves  us  closer  to  these  goals.  We  established  the  technical,  commercial,  and 
scientific  feasibility  of  our  approach  to  AR-based  adaptive  training. 

Technical  Feasibility 

In  terms  of  technical  feasibility,  the  Phase  I  research  efforts  related  to  Objective  1  (overcome 
technological  barriers  in  mapping  AR  content  to  physical  manikin)  and  Objective  2  (develop  AR  Paining 
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prototype),  established  the  merits  of  our  approach.  Early  feedback  from  military  clients  indicated  that  it 
was  vital  that  the  virtual  patient  and  physical  manikin  move  in  sync  for  effective  scenario-based  training. 
We  identified  and  tested  several  technological  solutions  to  this  requirement.  We  were  able  to  demonstrate 
that  the  technologies  could  be  incorporated  into  MART.  During  the  Phase  I,  we  also  designed  AR  assets 
and  adaptive  training  components  that  can  be  built  into  MART  during  the  Phase  II.  Certain  elements  were 
already  incorporated  into  the  prototype  -  specifically  the  virtual  patient  assets  associated  with  each  of  our 
training  scenarios.  Other  elements  (e.g.,  adaptive  training  components)  have  been  articulated  and  vetted 
against  the  existing  state  of  the  technology.  We  are  confident  that  we  can  build  them  into  MART. 

Commercial  Feasibility 

In  terms  of  commercial  feasibility,  during  the  Phase  I  effort  we  made  connections  to  both  military  and 
civilian  commercial  outlets.  We  have  an  existing  partnership  with  Innovative  Tactical  Training  Solutions 
(ITTS)  that  has  contracts  with  military  clients.  ITTS  has  a  marketing  and  sales  team  already  in  place  that 
have  been  trained  to  market  one  of  our  existing  products,  the  Virtual  Patient  Immersive  Trainer  (VPIT). 
ITTS  plans  to  continue  supporting  our  marketing  efforts  during  the  Phase  II.  Our  university  partner,  Ohio 
State  University,  has  also  expressed  interest  in  purchasing  a  version  of  MART  for  use  in  their  residency 
Paining  program.  OSU  has  a  state  of  the  art  simulation  facility,  and  is  interested  in  growing  their 
capabilities  to  include  emerging  technologies.  Outside  of  our  team,  the  director  of  the  simulation  center  at 
Cincinnati  Children’s  Hospital  Medical  Center  (CCHMC)  has  expressed  a  strong  desire  and  need  for  AR 
technology  in  the  CCHMC  simulation  center. 

Working  with  the  OSU  College  of  Medicine,  we  have  also  begun  to  explore  how  MART  can  be 
integrated  into  the  existing  curricula.  OSU  is  a  leader  in  medical  education.  Use  of  MART  for  airway 
management  and  tension  pneumothorax  training  will  position  OSU  to  serve  as  a  ‘best  practice’  model  for 
implementation  of  AR-based  training  to  other  academic  medical  centers,  sharing  lessons  learned  via 
traditional  dissemination  pathways.  We  have  three  physicians  on  our  team  for  the  Phase  II  who  will  assist 
in  creating  the  practice  model,  including  Dr.  Susan  Moffatt-Bruce,  executive  director  of  University 
Hospital,  who  will  serve  as  the  interested  customer  for  purchasing  the  technology  and  associated  training. 
OSU  represents  an  influential  partner  that  will  strengthen  our  commercialization  efforts. 

Scientific  Feasibility 

Based  on  our  review  of  the  relevant  literatures  and  the  expertise  of  our  team  members,  we  have  begun  to 
theorize  about  the  elements  of  AR  training  that  will  be  most  effective.  The  components  and  structure  of 
MART’S  adaptive  training  that  we  articulated  in  the  previous  sections  reflect  our  theoretical  framework. 
We  have  articulated  a  detailed  plan  for  assessing  the  strengths  and  limitations  of  individual  adaptive 
Paining  components  leveraging  the  AR  technology  in  MART  training.  In  the  Phase  II,  we  plan  to  test  key 
components  of  our  Paining  system  to  identify  research-  and  theory-based  design  principles  that  can  be 
applied  to  other  AR  training  applications,  resulting  in  a  Guide  to  Designing  Effective  AR  Adaptive 
Training. 

Summary 

During  this  Phase  I  effort,  we  successfully  advanced  the  state  of  AR  training  by  developing  innovative 
strategies  for  aligning  a  virtual  patient  with  a  physical  manikin.  We  established  important  partnerships 
with  ITTS  manikin  manufacturer  and  The  Ohio  State  University  College  of  Medicine  to  promote 
commercialization  in  both  military  and  civilian  health  sectors.  We  articulated  a  plan  for  evaluating  the 
impact  of  AR  on  learning  to  lead  to  scientifically-founded  principles  for  designing  AR-based  Paining.  We 
are  well-positioned  to  implement  technological  advances  identified  during  Phase  I  research,  conduct 
formative  and  summative  evaluations  as  part  of  the  development  cycle,  and  continue  down  the  path  to 
commercialization  in  a  Phase  II  project. 

5.  PUBLICATIONS,  ABSTRACTS,  AND  PRESENTATIONS 

We  presented  a  poster  (Appendix  B)  and  gave  a  demonstration  at  the  2018  Human  Factors  and 
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Ergonomics  Society  Healthcare  Symposium,  held  in  Boston,  MA  March  26-28,  2018.  Final  manuscripts 
are  due  by  April  23,  2018  for  inclusion  in  the  conference  proceedings.  The  funding  of  this  project  is 
acknowledged  in  all  publications.  We  will  send  the  finalized  manuscripts  to  the  sponsor  and  COR  when 
we  submit. 

Militello,  L.,  Sushereba,  C.,  Fernandez.  S.,  &  Patterson,  E.  (2018,  March,  in  press).  A  systematic  method 
for  using  critical  decision  method  interviews  to  generate  a  garden  path  scenario  for  human-in-the-loop 
simulations  for  training  and  evaluation  of  new  technologies.  Poster  session  presented  at  the  Human 
Factors  and  Ergonomics  Society  Healthcare  Symposium,  Boston,  MA. 

Militello,  F.  &  Sushereba.  C.  (2018,  March,  in  press).  Augmented  reality  for  recognition  skills  training. 
Demonstration  presented  at  the  Human  Factors  and  Ergonomics  Society  Healthcare  Symposium,  Boston, 
MA. 

6.  INVENTIONS,  PATENTS,  AND  LICENSES 

A  number  of  the  software  prototype  elements  developed  during  the  project  build  on  patent  filings  that  pre¬ 
date  the  Phase  I  project  (Application  Serial  No.  62/483,17 1).  No  new  patent  filings  were  made  during  the 
project  period.  However,  progress  on  the  adaptive  training  schema,  and  each  of  the  registration  prototypes 
(i.e.,  skeleton  tracking,  marker-based  tracking,  and  IMUs)  will  be  added  to  our  prior  filings.  An  updated 
filing  is  expected  on  or  before  April  6th,  2018. 

7.  REPORTABLE  OUTCOMES 

Progress  made  during  the  Phase  I  has  supported  improvements  to  our  commercial  software  development 
efforts.  The  VPIT  1 .0  software  launched  during  the  course  of  the  project  (see  Press  Release  in  Appendix 
C). 

8.  OTHER  ACHIEVEMENTS 

Nothing  to  report. 
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10.  APPENDICES 


Appendix  A:  Training  Scenarios  with  Expert  Models 
Available  Kit 


Combat  Medic 

•  CAT  tourniquet  (2) 

•  ETD  (2  -  6  in.) 

•  ETD  abdomen 

•  S -rolled  gauze  (4) 

•  Combat  gauze  (2) 

•  Tactical  comp  wrap 

•  Naso  (2  airway  and  lube) 

•  King  LTS-D 

•  Cyclone  BVM  packet 

•  Tactical  suction 

•  Tactical  CrikKit 

•  HyFin  (vent,  twin  pack) 

•  ARS  needle,  14g  (2) 

•  BOA  standard 

•  Saline  lock  kit  (4) 

•  Sharps  shuttle 

•  Splint  tactical  traction 

•  SAM  splint  (2) 

•  Trauma  shears  (large) 

•  7  hook  safety  cutter 

•  2  in.  Tape  (2) 

•  Case  Armadillo 

•  Nitrile  gloves  (8  pairs) 

•  PES  eye  shield 

•  Combat  casualty  card  (4) 

•  Casualty  reference  card 

•  Permanent  marker 


Figure  12.  NAR-4  Aid  Kit  from  North  American  Rescue  website: 
https://www.narescue.com/pre-hospital-care-ems- 

products/casualty-response-kits/nar-4-aid-kit 
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Tension  Pneumothorax 


Context:  An  explosive  ordnance  disposal  (EOD)  soldier  was  attempting  to  neutralize  a  small  improvised 
explosive  device  (IED).  The  IED  detonated  during  the  deactivation  attempt,  causing  a  blast  lung  injury, 
temporary  loss  of  consciousness,  blunt  force  trauma  to  the  rib  cage,  and  bums  to  the  patient’s  hands.  The 
patient  was  thrown  approximately  2  meters  away  from  the  blast  site,  and  landed  on  pavement. 

This  is  an  active  combat  situation,  so  key  concern  is  security.  Patient  is  a  male,  20-23  years  old, 
approximately  180  lbs. 

Time  1 :  30-60  seconds  after  POI 

Right  after  the  point  of  injury  (POI),  you  are  able  to  do  a  quick  assessment  of  the  patient,  but  you  must 
stay  alert  to  the  combat  conditions.  You  see  the  following: 

•  There  is  no  massive  hemorrhage  visible 

•  There  is  some  blood  in  the  patient’s  mouth 

•  The  patient  has  close  to  full  respirations.  The  inhalation  volume  is  decreased,  rate  of  breathing  is 
fast,  and  the  patient  is  having  difficulty  breathing. 

•  Patient  is  expressing  pain. 

•  Patient  asks,  “What  happened?” 

•  There  are  superficial  and  partial  thickness  burns  to  the  patient’s  hands. 

•  The  patient’s  bomb  suit  is  damaged,  smoldering,  and  stitching  is  tom  on  his  clothes. 


Your  assessment  is  interrupted  by  active  fire,  so  you  must  shift  focus  to  site  security. 


Expert  Model 

Important  cues: 

1.  Difficulty  breathing 

2.  Increased  rate  of  breathing 

3.  Decreased  volume  of  breathing 

4.  Mental  status 

5.  Patient  is  talking,  so  airway  is  patent 

6.  Lack  of  massive  hemorrhage 
Differential  diagnosis: 

1 .  Pneumothorax  -  either  regular  or  hemothorax 

2.  Pulmonary  contusion 

3.  Possible  airway  injury  (inhalation  burns) 

4.  Possible  head  injury 
Planned  actions: 

1 .  Secure  the  environment  and  the  patient 

2.  Position  the  patient  for  airway  management 

3.  Expose  the  patient  and  do  a  body  sweep 

4.  Check  trunk  for  any  lacerations  and  apply  occlusive  dressing 
Anticipated  patient  progression: 

1 .  Patient  will  deteriorate. 

Triage  level: 

Urgent  -  Cat  A 
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Time  2:  15  minutes  after  POI 


Your  team  has  secured  the  area,  and  you  are  able  to  continue  your  assessment  of  the  patient.  You  observe 
the  following: 

•  There  is  no  external  bleeding 

•  Blood  in  the  mouth,  and  now  you  notice  some  broken  teeth  (not  obstructing  the  airway).  It  looks 
like  the  patient  bit  his  cheek  and  tongue. 

•  You  remove  his  shirt  and  see  bruising  on  his  right  rib  cage,  near  the  nipple  line.  There  is  a  slight 
paradoxical  motion  to  his  right  lateral  chest  wall.  The  patient  has  increased  difficulty  and  effort  of 
breathing. 

•  Patient  is  talking  in  short  sentences  and  alert. 

•  The  patient’s  voice  is  hoarse  and  he’s  coughing. 

•  There  are  superficial  and  partial  thickness  burns  to  the  hands  and  face. 

You  are  now  able  to  move  the  patient  to  a  more  secured  area  where  you  can  administer  more  advanced 
treatments. 

Expert  Model 

Important  cues: 

1 .  Bruises  on  the  rib  cage 

2.  Paradoxical  chest  wall  motion 

3.  Flail  chest  segment 

4.  Difficulty  breathing 

5.  Patient  talking  in  short  sentences 
Differential  diagnosis: 

1 .  Pneumothorax  -  either  hemothorax  or  tension  pneumothorax 

2.  Flail  chest 

3.  Pulmonary  contusion 
Planned  actions: 

1 .  Assess  and  clear  the  oral  airway 

2.  Needle  decompression 

3.  Dress  wounds,  burp  any  chest  seals 

4.  Pain  management 
Anticipated  patient  progression: 

1 .  The  patient  will  decompensate  rapidly,  leading  to  death. 

Triage  level: 

Urgent  -  Cat  A 


Time  3:  45-60  minutes  after  POI 

The  patient  is  now  in  a  secure  and  protected  area.  You  re-assess  after  moving  the  patient,  and  observe  the 
following: 

•  No  external  bleeding 

•  The  patient  is  breathing  rapidly  and  wheezing.  Fie  is  becoming  hypoxic  and  cyanotic. 

•  The  paradoxical  motion  in  the  rise  and  fall  of  his  chest  is  more  pronounced,  and  now  you  see  a 
flail  chest  segment.  The  bruises  on  his  torso  are  larger  and  have  changed  color.  You  see  unilateral 
rise  and  fall  of  his  chest. 

•  The  patient  is  confused  and  his  skin  is  paler. 

•  Some  of  the  blisters  in  the  burned  area  to  the  hands  and  face  ruptured  while  the  patient  was  being 
moved  to  the  new  location. 
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Expert  Model 


Important  cues: 

1 .  Cyanosis 

2.  Shock,  pale  skin 

3.  Hypoxia 
Differential  diagnosis: 

1 .  Pneumothorax  -  tension  pneumothorax,  possibly  hemothorax 
Planned  actions: 

1.  Needle  decompression 

2.  Listen  to  the  lungs 

3.  Airway  management  (supraglottic) 

4.  Place  chest  tube 

5.  Give  blood,  fluid,  or  TXA 

6.  Oxygen 

Anticipated  patient  progression: 

1 .  The  patient  will  rapidly  decline  and  die. 

Triage  level: 

Urgent  -  Cat  A 


Time  4:  3  hours  after  POI 

The  patient  still  has  not  reached  a  definitive  care  facility,  but  he  is  sheltered  in  a  secure  location.  You  re¬ 
assess  him  and  notice  the  following: 

•  No  external  bleeding 

•  Patient  is  excessively  coughing  with  productive  red  sputum,  has  decreased  breath  sounds,  and 
there  is  blood  coming  out  of  his  mouth. 

•  You  see  unilateral  rise  and  fall  of  the  chest,  and  the  bruises  on  his  torso  have  worsened. 

•  He  shows  jugular  vein  distention  and  tracheal  deviation. 

•  His  skin  is  pale,  and  he  is  in  and  out  of  consciousness.  His  pupils  are  sluggish  to  respond  to  light 
when  his  eyes  are  open. 


Expert  Model 

Important  cues: 

1.  Jugular  vein  distention 

2.  Tracheal  deviation 
Differential  diagnosis: 

1 .  Tension  pneumothorax 
Planned  actions: 

1.  Needle  decompression 

2.  Continue  with  treatment  protocols 

3.  Assess  mentation 

4.  Listen  to  breathing 
Anticipated  patient  progression: 

1 .  Patient  will  die. 

Triage  level: 

Urgent  -  Cat  A 
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Airway  Obstruction 

Context:  An  RPG  strikes  the  driver  side  front  door  of  an  up-armored  vehicle  in  a  convoy.  The  driver 
(wearing  a  ballistic  helmet,  body  armor,  and  gloves)  was  closest  to  the  explosion  and  remained  entrapped 
in  the  vehicle  for  approximately  90  seconds  before  his  teammates  are  able  to  pull  him  from  the  burning 
wreckage.  He  receives  partial  and  full  thickness  burns  around  his  neck,  jaw,  and  upper  cheeks.  The  body 
armor  protected  his  chest  from  any  direct  bums  or  penetrating  injury.  The  soldier’s  upper  extremities 
were  hit  with  shrapnel  and  debris.  His  gloves  protected  his  hands,  but  there  are  ring  burns  around  his 
wrist. 

Time  1 :  Immediately  following  POI 

You  are  traveling  with  the  security  convoy  and  are  able  to  get  to  the  patient  shortly  after  he  was  removed 
from  the  burning  vehicle.  After  moving  the  patient  to  a  secure  location,  you  make  the  following 
observations  during  your  initial  assessment  of  the  patient: 

•  There  is  no  massive  hemorrhaging  noted. 

•  There  is  blood  on  his  uniform  from  lacerations  to  his  arms,  but  none  of  the  injuries  have  created 
arterial  bleeding. 

•  The  patient  has  singed  nostril  hairs  and  reddening  of  the  skin  on  his  face.  There  are  also  small 
pieces  of  shrapnel  in  his  face.  His  lips  are  red,  like  they’ve  been  sunburned.  There  are  no  major 
secretions  in  his  mouth  and  his  neck  looks  normal. 

•  His  voice  is  hoarse  and  he’s  coughing.  He  has  increased  respirations.  There  is  bilateral  rise  and 
fall  of  his  chest,  and  normal  expansion  and  contraction  of  his  rib  cage. 

•  Patient  is  conscious  and  in  pain.  He  is  speaking  in  coherent  words,  but  he  is  rambling  and  does 
not  appear  to  be  fully  aware  of  what  happened.  He  looks  dazed  and  stressed. 

•  Carotid  and  radial  pulses  are  the  same. 

•  The  patient’s  skin  feels  warm/hot  to  the  touch. 

•  He  is  sweaty,  and  has  light  burns  in  addition  to  the  lacerations  on  his  arms. 

Expert  Model 

Important  cues: 

1.  Patient’s  voice  is  hoarse 

2.  Singed  nostril  hairs 

3.  Skin  on  patient’s  face  is  red  and  burned 

4.  Respirations  are  fast 
Differential  diagnosis: 

1 .  Thermal  injury  to  the  airway 

2.  Possible  TBI 
Planned  actions: 

1.  Maintain  security 

2.  Position  patient  to  manage  the  airway 

3.  Assess  the  patient’s  mouth  for  swelling,  carbonaceous  sputum,  and  other  secretions; 
suction  and  clear  the  mouth 

4.  Expose  the  patient  and  do  a  full  body  sweep  for  other  injuries 
Anticipated  patient  progression: 

1 .  Patient  is  sick  -  his  airway  is  significantly  compromised,  which  will  lead  to  respiratory 
distress  and  death  if  not  managed. 

Triage  level: 

Urgent  -  Cat  A 
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Time  2:  5-10  minutes  after  POI 


The  patient  is  lethargic  and  in  obvious  respiratory  distress,  so  you  reassess  his  condition.  You  notice  the 
following  on  your  reassessment: 

•  No  additional  bleeding. 

•  Tissue  along  the  airway  is  beginning  to  swell,  and  the  patient  is  coughing  up  thick  carbonaceous 
sputum  with  some  yellow-green  phlegm  and  blood.  The  cough  is  productive. 

•  There  is  some  bleeding  inside  the  mouth  from  the  patient  biting  his  cheek. 

•  The  redness  on  the  skin  around  the  patient’s  mouth  is  darkening.  Blisters  are  forming  across  the 
patient’s  cheeks  and  forehead. 

•  The  nostrils  are  becoming  smaller. 

•  There  is  general  swelling  around  the  patient’s  face  -  his  lips  and  tongue  are  becoming  larger. 

•  There  seem  to  be  more  secretions  in  his  mouth  (from  swelling  and  inability  to  swallow),  and  the 
oral  pharynx  is  starting  to  close  up,  becoming  more  voluminous  and  wet.  The  secretions  are  light 
to  dark  gray  with  black  specks. 

•  The  patient’s  pulse  ox  is  86%. 

Expert  Model 

Important  cues: 

1 .  Patient  is  coughing  up  carbonaceous  sputum 

2.  Airway  is  swelling 

3.  Nostrils  are  smaller  and  the  face  is  swelling 
Differential  diagnosis: 

1 .  Airway  obstruction  due  to  thermal  burns  and  edema 
Planned  actions: 

1 .  Advanced  airway  -  cric  or  intubate 

2.  Oxygen/ventilate 
Anticipated  patient  progression: 

1 .  Airway  obstruction  will  lead  to  death 
Triage  level: 

Urgent  -  Cat  A 


Time  3:15  minutes  after  POI 

Non-surgical  attempts  at  managing  the  patient’s  airway  have  failed.  The  patient  is  worsening,  and  you 
observe  the  following: 

•  Tissue  in  the  nose  and  mouth  is  significantly  swollen.  The  patient’s  lips  are  triple  in  size. 

•  Larger  blisters  are  forming  on  the  patient’s  face. 

•  Carbonaceous  sputum  and  secretions  are  very  prominent. 

•  You  hear  stridor. 

•  The  patient  is  wheezing,  and  you  hear  wet  lung  sounds.  There  is  an  increased  effort  of  breathing 
and  an  increased  rate  of  breathing.  There  is  decreased  tidal  volume. 

•  The  patient  is  fatigued. 

•  The  patient’s  pulse  ox  is  76%. 

•  The  patient  has  a  muted  cough. 
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Expert  Model 


Important  cues: 

1 .  Stridor/wheezing 

2.  Airway  swelling  is  worse 

3.  Hypoxia 
Differential  diagnosis: 

1 .  Impending  respiratory  failure  from  airway  obstruction  due  to  thermal  burns  and  trauma 
Planned  actions: 

1 .  Cric  or  other  type  of  surgical  airway 

2.  Oxygen/ventilation 
Anticipated  patient  progression: 

1 .  Respiratory  failure  and  death 
Triage  level: 

Urgent  -  Cat  A 
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Appendix  B:  Poster  Submission  to  HFES  Healthcare  Symposium 
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Appendix  C:  VPIT  1.0  Press  Release 


FOR  IMMEDIATE  RELEASE: 
CONTACT: 

Contact  Person:  Christen  Sushereba 
Company  Name:  Unveil,  LLC 
Voice  Phone  Number:  937-602-2421 
Email:  christen@unveilsystems.com 
Website:  http://unveilsystems.com/ 


Unveil  Announces  Release  1.0  of  Virtual  Patient  Immersive  Trainer 

Cincinnati,  Ohio,  March  15th,  2018  -  Unveil,  a  healthcare  training  technology  company  in  Cincinnati, 
Ohio  has  announced  the  first  release  of  their  augmented  reality-based  training  software  designed  for 
emergency  combat  medicine.  The  Virtual  Patient  Immersive  Trainer,  or  VPIT,  enhances  the  realism  of 
emergency  medical  conditions  such  as  tension  pneumothorax,  massive  hemorrhage,  and  airway 
obstruction. 

The  VPIT  works  in  conjunction  with  a  physical  training  manikin  to  support  the  development  of  both 
recognition  and  treatment  skills.  The  software  is  a  companion  to  the  Innovative  Tactical  Training 
Solutions’  industry-leading  TOMManikin.  The  VPIT  software  includes  scenario-based  instruction  for 
combat  medicine  students,  as  well  as  coaching  and  audit  tools  for  instructors.  Wearing  a  mixed  reality 
headset,  students  see  a  realistic  virtual  patient  whose  appearance  adaptively  changes  based  on  the 
treatment  path  chosen  by  the  student  or  trainee.  Students  make  real  physical  interventions  on  the 
TOMManikin  which  result  in  changes  to  both  the  physical  and  the  virtual  patient.  Instructors  control  the 
flow  of  the  scenario  and  receive  reports  that  summarize  student  performance  to  improve  after  action 
review  sessions. 

The  entire  VPIT  1.0  augmented  reality  experience  is  delivered  through  the  Microsoft  HoloLens.  The 
software  comes  pre-loaded  on  the  HoloLens  hardware  and  includes  the  Tactical  Combat  Casualty  Care 
(TCCC)  module. 

“We  are  extremely  excited  to  release  VPIT  1.0,”  said  Unveil  Co-Founder  and  CEO  Steve  Wolf.  “We 
think  that  our  integration  with  the  TOMManikin  is  truly  unique.  It  builds  on  the  value  of  ITTS’s  existing 
simulation  manikin  by  bringing  cutting-edge  realism  to  the  diagnostic,  triage,  and  treatment  training 
experience.” 

“Now  instructors  can  use  these  AR-enhanced  manikins  to  train  students  not  only  on  how  to  perform 
medical  interventions,  but  also  how  to  recognize  the  subtle  physical  signs  and  symptoms  of  dangerous 
conditions,”  said  Unveil  Co-Founder  and  Combat  Rescue  Officer,  Oliver  Smith. 

The  VPIT  is  available  for  purchase  through  the  ITTS  website.  For  additional  information  please  visit 
www.tommanikin.com 

About  Unveil,  LLC 

Unveil  is  a  training  technology  company  located  in  Cincinnati,  Ohio.  Unveil  was  founded  in  2016  and 
provides  augmented  reality-based  training  systems  for  emergency  medical  providers  in  the  commercial 
and  military  industries. 

For  more  information,  contact  Christen  Sushereba  -  christen@unveilsystems.com 
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Appendix  D:  List  of  Paid  Personnel 


The  following  personnel  were  paid  on  this  project: 

Unveil,  LLC: 

•  Laura  Militello,  Principal  Investigator 

•  Christen  Sushereba,  Research  Associate 

•  John  Hendricks,  Creative  Technologist 

•  Oliver  Smith,  PJ,  Subject  Matter  Expert 

•  Julie  Dilulio,  Visual  Designer 

•  Jason  Van  Cleave,  Software  Programmer 

•  Steve  Wolf,  Commercialization  Lead  ( unpaid  in  kind  effort) 

Ohio  State  University: 

•  Emily  Patterson,  Co-Principal  Investigator 

•  David  Bahner,  MD,  Subject  Matter  Expert 

•  Simon  Fernandez,  Research  Assistant 

•  Jacob  Socha,  Research  Assistant 

Stonier  Henke  Associates,  Inc.: 

•  Sowmya  Ramachandran,  Adaptive  Training  Consultant 

Innovative  Tactical  Training  Solutions: 

•  Keary  Miller,  Commercialization  Partner 
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